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Executive Summary 



This report presents the results of the interim evaluation of the National Literacy Program 
(NLP) conducted between July 2005 and December 2006. 

The purpose of this interim evaluation was to acquire information to help strengthen or 
improve the overall performance and outcomes of the NLP. The evaluation examined issues 
related to: 

• Rationale and Relevance; 

• Implementation: 

— Program Utilization and Reach; 

— Program Implementation, Efficiency and Effectiveness; and 

— Performance Measurement. 

• Success: 

— Achievement of NEP Objectives; 

— Initial Eearner Impacts; and 

— Initial Capacity Building. 

In addition, the interim evaluation was intended to: 

• Determine whether sufficient data was being collected to inform the summative evaluation 
and identify opportunities for improvement to fill any potential gaps; 

• Assess whether the changes brought to the logic model following the 2002 evaluation, 
including the clarification of the program definitions, objectives, and targets, made 
achievement of NEP objectives more measurable; and 

• Assess whether new data collection requirements and methodologies have been put 
into place since the 2002 evaluation to ensure that both short and long-term outcomes, 
including changes made since the 2002 summative evaluation, can be measured. 

Einally, the interim evaluation assessed NEP performance in relation to commitments made 
in the National Eiteracy Secretariat (NES) Management Response to the 2002 summative 
evaluation, including: 

• Increasing the transparency of funding decisions; 

• Increasing the accessibility of projects and project results; 

• Simplifying the funding application process; 
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• Increasing the reliability and credibility of local research projects; 

• Increasing the usage of contributions; and 

• Developing a framework for a Pan-Canadian Literacy Strategy. 

Program Background 

The NLS was formed in 1988 to ensure that Canadians have opportunities to develop and 
improve the ever-expanding literacy skills needed to function at work, at home and in the 
community, and to support the development and dissemination of information on literacy 
issues to increase awareness and understanding. 

The NLS did not deliver programs directly, but provided financial and technical assistance 
for activities that promoted its goals. The grants and contributions program, administered 
through the NLP, was the main instrument used to engage key partners in literacy projects. 

The NLP’s annual grants and contributions budget ranged from $27.5 million to 
$38.4 million between 2002 and 2005. NLP funding was provided for five categories of 
eligible activities: Learning Materials, Access and Outreach, Coordination and 
Information, Public Awareness, and Research. 

On April 1, 2006, Human Resources and Social Development Canada’s (HRSDC) three 
existing adult learning and literacy grants and contributions programs (The National Literacy 
Program, the Office of Learning Technologies and the Learning Initiatives Program) were 
integrated under the Adult Learning, Literacy and Essential Skills Program (ALLESP). 
This integrated program enhances HRSDC’ s capacity to respond to the needs of its key 
partners and communities across Canada in the area of adult learning, literacy and essential 
skills. The integration is expected to result in efficiency and effectiveness gains through 
improved co-ordination of program activities, streamlined procedures and better allocation of 
resources. The current interim evaluation of the NEP was designed prior to the creation 
ofAEEESP. The evaluation methodology was therefore not affected by this change. 
However, results from the current evaluation will no longer lead to a summative evaluation 
of the NEP but will be used to inform a future evaluation of AEEESP. 

Evaluation Scope and Methodology 

This evaluation focused on the fiscal years 2002-2003, 2003-2004 and 2004-2005 of the 
program. It is based on multiple lines of evidence including: an administrative data review, 
preliminary key informant interviews with NEP representatives (n=7), a review of documents 
and literature, an online survey of NEP funding recipients (n=191), project case studies 
(n=20), organisation case studies (n=6), a province/territory case study in Prince Edward 
Island, an online survey of NEP project end-users (n=69), an expert panel that examined 
a sample of eleven NEP-funded products, and phone interviews with unfunded organisations 
(n=2). An iterative approach was taken to integrate the methodologies so that each data 
collection activity would build upon the others. 
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Key Findings and Conciusions 



Rationale and Relevance 

Program Consistency with Needs and Priorities: Evidence shows that the NLP’s support 
to developing literacy awareness, programming and access across Canada is compatible with 
both the federal government’s and HRSDC’s priorities pertaining to inclusive participation 
of all Canadians in our new learning society and knowledge-based economy. 

While the program does not address the needs expressed by literacy stakeholders for 
long-term core funding of literacy programming, it provides funding for coordination and 
networking, awareness and promotion and research, areas literacy stakeholders identified 
as needing federal government support. The allocation of NLP funds is compatible with the 
varied needs of the literacy community given its responsive partnership approach to setting 
funding priorities and the flexible definition of its five activity areas. Also, there appears to be 
few alternative sources of funding, in the areas examined, for literacy capacity building, 
coordination and program development. As such, the NLP plays a unique and necessary role. 

Relevance of Federal Government Role in Literacy: Overall, evidence shows that a need 
remains for federal government intervention in the field of adult literacy, particularly 
as an initiator, promoter and supporter of national initiatives and innovative partnerships 
among various stakeholders. The NLP’s contribution to literacy appears to be compatible 
and complementary to the role played by provinces, this complementarity being ensured 
by the effective partnership approach adopted. In this sense, this program does not appear 
to be a candidate for realignment with the provinces. 

Implementation 

Program Utilization and Reach: Some evidence shows that the NLP’s direct interventions 
reach a wide variety of stakeholders, particularly literacy organisations, non-profit service 
organisations and indirectly through funded project outputs, literacy practitioners and learners, 
including youth, families, and seniors from various Canadian provinces. The program has 
taken a responsive approach^ to addressing local literacy needs and, as such, did not set 
specific targets for the reach of its activities. Lurthermore, no data is being collected by the 
program on the actual end-users of NLP-funded activities, thereby preventing an accurate 
assessment of its reach. 

Program Implementation, Efficiency and Effectiveness: Program eligibility criteria are 
clearly outlined in the application package and neither NLP representatives nor funding 
recipients expressed having particular issues with this component of program implementation. 



* Responsive approach means that the NLP has made funding decision in reaction or in response to needs and priorities 
expressed by the stakeholders rather than based on set priorities and criteria. 
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NLP’s approach to determining priorities and distributing funding also appears to be 
appropriate for achieving its intended immediate outcomes given that it funds projects in each 
of its five target categories using project selection criteria that are based on local needs and 
priorities, and established in collaboration with each province/territory. NLP and PEI 
government representatives find this approach appropriate, given jurisdictional realities. 
Funding recipients from organisations case studies were, however, generally unaware of the 
process, thereby indicating that the program could be more explicit in its information 
regarding selection of projects. In the same way, only 57% (out of 191) of funding recipient 
survey respondents expressed satisfaction with the transparency of the proposal review 
process and 54% with the timeliness of the funding approval process. 

Performance Measurement: Although the NLP collects information that could be used 
to assess the performance of the projects it funds, it does not currently have a specific set 
of performance measures against which the results of its activities and funded projects could 
be measured, thereby limiting NLP’s performance measurement and reporting capacity. 
Furthermore, the program does not have a system in place to systematically store and use 
project performance information. In the absence of such a system, the NFP has limited ability 
to measure and report on whether it is achieving its objectives. The decision not to pursue 
initial steps to implement a performance measurement system was attributed in part to Senior 
Management’s desire to focus on the new AFFESP, for which a performance measurement 
system is being developed. No evidence was found that performance evidence is being used 
for program or management decision-making purposes on an ongoing basis. 

Success 

Achievement of NLP Objectives: Evidence shows that the NFP is contributing to meeting 
its objectives of supporting the production and utilisation of quality and useful literacy tools 
and materials, improvements in accessibility and dissemination of literacy services and 
resources, the application of literacy best practices, and the creation of strategic partnerships. 

NFP project outputs included literacy tools and materials, research reports, partnerships with 
other organisations, and awareness raising and promotional events and products. These were 
generally considered good quality and highly useful by practitioner end-users, while some 
products, especially research reports, were seen by literacy experts to be of low quality and 
relevance. 

NFP-funded products are being used by a wide variety of individuals, including literacy 
professionals and learners, for a wide range of purposes including improving current literacy 
training practices and developing new literacy training programs and tools. Strategic 
partnerships, for leveraging additional funding and/or for sharing expertise and resources, are 
also a key feature of all projects examined. 
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There was weaker evidence of NLP-funded activities contributing to increased public 
awareness and learner participation, in part due to the absence of data on the number and 
profile of project end-users. Some limitations were also found in the NLP’s mechanisms 
to promote accessibility and dissemination of tools and materials. While the program 
expects projects to disseminate their project results and encourages posting of funded 
products on the National Adult Literacy Database (NALD) and various provincial literacy 
websites, this is not systematically done and the NLP does not have a formal mechanism for 
rolling up the lessons learned and best practices developed by its funded projects. 

Initial Learner Impacts: In the absence of specific data on the learners who benefited 
from NLP-funded products, anecdotal evidence was collected from funded organisations 
and from literacy practitioners who witnessed first hand the positive impacts of these 
products on a variety of individuals. While not every project examined entailed impacts 
on learners, some examples recorded include improvement of studying and literacy skills 
among youth and increased motivation to return to school, which led to completion of 
high school or school credits for several as well as increased confidence in learning and 
self-esteem gained by learners as part of their literacy activities. 

Initial Capacity Building: Evidence was collected of the NLP’s contribution to capacity 
building. The NLP’s support to national literacy organisations, its influence in promoting 
new emerging areas such as workplace and family literacy, and its support and 
encouragement for the creation of strategic partnerships have been recognized by 
stakeholders in all categories. A majority of funding recipients surveyed also reported that 
their project would not have been implemented without NLP funding. Examples of positive 
impacts of NEP funding on funded organisations include the ability to secure long-term and 
ongoing funding and support from other sources and increased visibility, credibility and 
recognition of the organisation at the provincial and national levels. Impacts of funded 
projects on literacy practitioners include increased capacity to serve their clientele, most 
notably through partnership building and networking opportunities, as well as increased 
access to relevant training tools and best practices. 

Targeting of Families in Official Language Minority Communities: Sixteen projects 
were funded under the Action Plan for Official Eanguages (APOE) Eamily Eiteracy 
Initiative between 2003-2004 and 2005-2006. However, it remains too early to draw 
conclusions on the success of this initiative, given that funded projects remain in the early 
stages of their implementation. Project descriptions indicate that funding was targeted 
towards the development of networks, organisational and programming capacity, 
research, and promotion of family literacy in each of the provinces and at the national 
level through the work of the ECAE . This is consistent with the focus intended by the 
Eamily Eiteracy Initiative. 



^ NALD serves as a resource for technological development in literacy and has developed an internet-based information 
service for literacy policy and practice. 

^ Federation canadienne pour I'alphabetisation enfrangais. 
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other Findings and Conciusions 



Whether Sufficient Data Was Collected to Inform an Eventual Summative Evaluation: 
An eventual summative evaluation would benefit from data on the products and, if still 
desirable, on end-users of the projects funded by the program. This data was not 
systematically collected and recorded by the program. 

Changes to Logic Model: Intended revisions have been made to the program logic 
model, better reflecting the linkages between its activities and expected outcomes and 
proposing outcomes that better reflect the program’s levels of accountability. However, 
in order to fully meet the intent of these changes, which is to make achievement of NLP 
objectives more measurable, the program would have needed to approve performance 
indicators and targets for each expected outcome. 

Requirements and Methodologies in Place to Ensure Measurability of Outcomes: No 
system is in place to ensure the measurability of program outcomes. This system would have 
required detailed performance measures and a mechanism for systematically collecting 
information on project outputs and, if possible and desirable, on end-users. 

More Transparent Funding Decisions: Feedback collected from funding recipients 
indicates that the process for making funding decisions is unknown to many and an aspect of 
the program’s implementation garnering one of the lowest satisfaction ratings. 

Increasing the Accessibility of Projects Results: While the NLP has put in place a number 
of measures to increase accessibility of project results, posting of project outputs on NALD 
and other provincial websites is not systematically done. The NLP does not have a formal 
mechanism for rolling up the lessons learned and best practices developed by its funded 
projects. 

Simplifying the Funding Application Process: Improvements were brought to the NLP 
application process, namely the posting of application forms and instruction on the 
NLP website, which appear to have addressed a majority of funding recipients’ concerns in 
this respect. However, 14% (out of 191)"^ of surveyed funding recipients remain dissatisfied 
or very dissatisfied with the simplicity of the funding application process. 

Steps Taken To Increase the Reliability and Credibility of Local Research Projects: 
Although a study was commissioned by the NLP to examine the research reports funded 
between 1998 and 2003, no evidence was found that the NLP has implemented changes to its 
approach in funding research projects to increase their reliability and credibility. 
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Number in bracket following a survey result in percentage represents the total number of respondents for this specific 
question. 
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Increasing the Usage of Contributions: The NLP has established an informal guideline 
for contributions to be used for funding national organisations (except for the National 
Indigenous Literacy Association) and for projects of $350,000 or more, with the aim of 
reaching a proportion of 22% contributions. The proportion of contribution projects has 
increased to 11% in 2004-2005, while the proportion of contribution funding 
has increased to 22%. However, there is no evidence that the shift operated towards 
increased use of contributions has had the intended effect of increasing project 
accountability for results. It has however increased the administrative burden on NLP 
staff and funding recipients, while increasing the length of the project approval process. 

Developing a Framework for a Pan-Canadian Literacy Strategy: A framework for 
a pan-canadian literacy strategy was presented in 2005 as part of the report “Towards a Fully 
Literate Canada. Achieving National Goals through a Comprehensive Pan-Canadian Literacy 
Strategy,”^ thereby addressing NLS commitments. No evidence was found, however, that the 
proposed framework or any of the reports’ recommendations were to date adopted or 
implemented. 



^ Presented by the Advisory Committee on Literacy and Essential Skills to the Minister of State for Human Resources 
Development in November 2005. 
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Management Response 



Introduction 

An interim evaluation of the National Literacy Program (NLP) was conducted between 
July 2005 and December 2006. The purpose of this interim evaluation was to acquire 
information to help strengthen or improve the overall performance and outcomes of the NLP. 

More specifically, the evaluation considered the NLP’s relevance, implementation and 
levels of success. It also sought to determine the extent to which the NLP had met the 
commitments made following the 2002 evaluation: increasing the transparency of 
funding decisions; increasing the accessibility of projects and project results; simplifying 
the funding application process; increasing the reliability and credibility of local research 
projects; increasing the usage of contribution funding relative to grants; and developing 
a framework for a Pan-Canadian Literacy Strategy. 

While the evaluation was underway, NLP underwent a significant change. On April 1, 2006, 
the NLP, the Office of Learning Technologies (OLT) and the Learning Initiatives Program 
(LIP) were integrated into a single cohesive program, the Adult Learning, Literacy and 
Essential Skills Program (ALLESP). The ALLESP was created to enhance focus, strengthen 
capacity and improve results measurement in HRSDC’s adult learning and literacy activities. 

In line with the September 25, 2006 effective spending measures announcement, the 
ALLESP re-targeted its investments towards national priorities and areas of clear federal 
responsibility. On April 11, 2007, as part of measures to support the objective of 
becoming a Centre of Excellence for the development and delivery of human resources 
and social development policies and programs, the National Office of Literacy Learning 
(NOLL), which administered the ALLESP was moved from the Learning Branch to the 
new Skills and Employment Branch. A new Office of Literacy and Essential Skills 
(OLES) was created which brings together the Essential Skills Initiative (ESI) 
and NOLL. OLES will provide a national knowledge base of best practices, models and 
applied research, as well as practical tools and instruments that support and strengthen 
partnerships in literacy and essential skills programs. OLES will also have responsibility 
for Grants and Contributions programming, including ALLESP. 

Findings and Responses 

Overall, the NLP was found to be compatible with both the federal government’s and 
HRSDC’s priorities with respect to the inclusive participation of all Canadians in the 
knowledge-based economy. The NLP provided funding for areas literacy stakeholders 
identified as needing federal government support, such as coordination and networking, 
awareness and promotion, and research. Anecdotal evidence collected from funded 
organizations and literacy pratictioners shows learners benefitted from the program 
by strengthening or developing new skills, becoming motivated to further their learning 
and improving their ability to function effectively in society. There also exists strong 
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evidence of the NLP’s contribution to capacity building in the literacy field, notably for 
funded organizations and literacy practitioners. With few alternative sources of funding, 
the NLP plays a unique and necessary role in literacy in Canada. 

The key evaluation findings and respective responses to each are laid out below. In some 
cases, the OLES has initiated actions within the ALLESP that respond to the findings, in 
other cases, the evaluator’s recommendations have been overtaken by wider departmental 
or governmental events. 

Relevance 

1. Program Consistency with Needs and Priorities: The NLP was found to be consistent 
with both the needs and the priorities of the literacy community and the Government 
of Canada. 

The AEEESP continues to respond to these priorities in its capacity as a program that 
supports the development of literacy and essential skills. Eiteracy and essential skills are 
key to building Canada’s Knowledge Advantage, which aims to create the best-educated, 
most-s ki lled and most flexible workforce in the world. 

AEEESP funding is allocated to projects that contribute to the development of 
literacy and essential skills in and for the workplace, and that help Canadians develop 
the literacy skills they need for all aspects of family and community life. Eollowing 
the Government of Canada’s commitment in the 2006 Speech from the Throne to 
improve opportunity for all Canadians, the 2007 AEEESP Call for Proposals gives 
priority to projects that have as their subject Aboriginal Canadians, immigrants. 
Official Eanguage Minority Communities and/or lower-skilled workers. 

The OEES will become a “centre of expertise” to improve literacy and essential skills for 
Canadians. To complement AEEESP funding, OEES will work to address the literacy 
and essential skills needs of Canadians in areas of direct federal responsibility by building 
on federal strengths and complementing the efforts of others. 

2. Relevance of Federal Government Role in Literacy: The evaluation found that NLP filled 
the need for federal government intervention in the field of adult literacy, as an initiator, 
promoter and supporter of national initiatives and innovative partnerships. In addition, it 
was found that the NLP’s role in this field was compatible and complementary to that of 
the provinces and territories. 

Continuing this national role in literacy, the AEEESP creates opportunities for adults 
to improve their literacy and essential skills. Projects under the AEEESP will be 
results-oriented and will focus on literacy and essential skills as they build the knowledge 
base, develop support mechanisms or carry out outreach activities to prepare people for 
work or improve the health of families and vitality of communities. 



X 
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Implementation 



3. Program Utilization and Reach: NLP direct interventions were found to reach a variety 
of stakeholders. Because the program was guided by stakeholders in defining and 
addressing literacy problems, it did not establish targets at the outset. 

It is necessary for a literacy program to be needs-based and, as such, ALLESP policies 
and programs will continue to be developed based on research and intelligence received 
outlining the needs of the literacy community. The research OLES and its stakeholders 
conduct on the needs of Canadians will now, however, be used to inform the program’s 
targets. A new performance measurement framework is being developed that will 
quantify targets. 

The AEEESP also funds projects that build sustainable networks and partnerships among 
literacy and essential .skills stakeholders, maximizing the engagement of learners and 
extending the program’s reach. 

4. Use of Contributions Agreements Relative to Grants: The NLP increased its use of 
contribution agreements but this did not have the desired effect of increasing NLP 
project accountability for results because no clear targets were established. Instead, 
it increased the administrative burden on NLP staff and funding recipients, while 
increasing the length of the project approval process. 

It is recognized that the project approval process is longer for contribution agreements 
than for grants. In addition, contribution agreements require funding recipients to submit 
quarterly financial and activities reports. While more monitoring is required during 
contribution projects, the additional work required will facilitate ongoing improvements 
in the quality of projects, greater accountability and results measurement. In order to 
foster improved results measurement with less administrative burden, a new reporting 
template is being developed by the OEES for projects resulting from the 2007 call for 
proposal (CEP). The information collected will be used to assess the performance of the 
wider AEEES program. 

5. Program Implementation, Efficiency and Effectiveness: Program eligibility criteria 
were found to be clearly outlined in the application package. It was also found that 
the program ’s six short-term aims were being met as a result of NLP ’s funding 
approach. However, the NLP’s project selection processes were found to be unknown 
to funding recipients. Funding recipients in the organisation case studies indicated 
that they were unaware of the way NLP funding priorities are determined and 
funding distributed. Similarly, only 57% of survey respondents were satisfied with the 
transparency of the proposal review process. 
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These issues have been addressed in documentation for the 2007 ALLESP Call for 
Proposals, which provides a description of the project selection process from receipt 
of funding application to Ministerial approval, and provides examples of selection 
criteria to be considered by the review committee. In addition, the CEP contains 
aPrequently Asked Questions (EAQ) section which is regularly updated to reflect 
questions received. The CEP documents contain a toll-free number and an email link 
should potential applicants have further questions. 

Before the CEP was launched. Senior Management at OEES held conference calls with 
provincial and territorial (P/T) government counterparts, P/T literacy coalitions and 
national hteracy organizations to introduce the new OEES and its priorities, to explain 
the new CEP and to provide an opportunity for these groups to pose questions. During the 
course of the CEP, OEES officials visited each P/T and P/T partners and coalitions were 
encouraged to invite their key stakeholders to participate in the meetings and raise any 
issues they may have. 

6. Performance Measurement: The NLP did not have a performance measurement 
framework in place, and performance information was not used to inform management 
decision making processes. 

The reason this was not done was because of the shift in focus to the development of the 
soon to be launched AEEESP, for which a performance measurement system would 
be developed. 

An initial performance measurement and evaluation framework was developed for the 
AEEESP, prior to the launch of the program in 2006. This framework is currently being 
updated to reflect the revised expected results of the program, subsequent to the 
April 2007 merger with ESI. The revised framework will be implemented in 2007-2008. 

Success 

Achievement of NLP Objectives 

7. Reliability and credibility of research projects: While a majority of NLP-funded 
project outputs were considered good quality and highly useful by end users 
consulted as part of the case studies, some products were seen by experts to be low in 
quality and relevance. Little evidence was found to suggest that the NLP took steps to 
increase the reliability and credibility of local research projects, despite Management 
Commitments made in response to the previous evaluation. 

The OEES is beginning to synthesize existing research on literacy and essential .s ki lls 
issues faced by four key HRSDC target groups: Aboriginal Canadians, immigrants, 
official language minority communities and lower s ki lled workers. The results will be 
used to help ensure that future AEEESP-fimded research projects support activities that 
address identified gaps, and contribute to cumulative knowledge development at the 
national level. 
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The evaluation found that NLP-funded research ran the risk of duplicating work done 
in other regions/communities and due to insufficient program monitoring, projects lacked 
the rigour necessary to be considered applied research. With the integration of multiple 
programs under the OLES, rigorous assessment and accountability is already in greater 
practice for ALLESP projects. As of April 11, 2007, all funding proposals submitted 
within the branch will be reviewed by a Director General-level internal review 
committee. This will help to ensure rigour in research projects. 

To further encourage the reliability and credibility of funded projects, all project 
proposals submitted under an AEEESP Call for Proposals must include a citation of 
relevant literature to illustrate that the proposed project does not duplicate work that has 
already been done, to explain how it will build on previous work in an important and 
original way and to show that the applicant is fa mi liar with the topic to be undertaken. 

Eurthermore, to help to direct and ensure the relevance of OEES research and policy 
development and to reduce duplication of research, national organizations and P/T 
coalitions are being asked to provide annual reports offering environmental scans, 
information on emerging issues and research being carried out across the country. 
This information will be provided through an agreement as a deliverable for multi- 
year operational contribution funding. 

8. Insufficient collection of end-user data: It was found that the NLP did not collect 
data on end users, which prevented an accurate assessment of the program ’s reach 
and of the extent to which NLP-funded activities contributed to increased public 
awareness and learner participation. 

Given that the mission of the NEP was to “ensure that Canadians have opportunities to 
develop and improve the literacy skills needed to function at work, home and in the 
community”, end-user data would have facilitated measurement of the program’s 
success. The NEP, however, only had indirect access to project end-users and project 
funding recipients were often unwilling to disclose end-user contact information for 
confidentiality reasons. The Office of Teaming Technologies (OET) was involved in 
a feasibility study, under Phase I of the program’s Summative Evaluation, to explore the 
collection of this type of data. Those involved encountered numerous difficulties, either 
because projects sponsors did not collect the information or did not want to share it for 
confidentiality reasons. 

The AEEESP will play an indirect role in helping adults improve their literacy and 
essential skills. The focus of the AEEESP will be on funding activities that build the 
knowledge base, develop effective support mechanisms and help build sustainable 
partnerships and networks that actively improve literacy and essential skills, rather than 
on direct provision of services to individuals. The recommendation on the collection of 
end-user data is no longer relevant in the context of the new focus of the AEEESP. 
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9. Lack of a formal mechanism for dissemination: It was noted that while the NLP 
expected projects to disseminate their project results and encouraged posting of 
funded products on NALD and other websites, this was not systematically done 
and the program did not have a formal mechanism for rolling up the lessons learned 
and best practices developed by its projects for the benefit of all stakeholders. 

Putting the tools and knowledge generated by projects into the hands of those who need 
them is one of the three key business lines of the OLES. OLES is developing a website 
on which will be posted tools developed for ALEESP stakeholders, AEEESP project 
descriptions and findings, research results, and news related to literacy and essential skills 
issues. An NEP project database is also being developed to be posted on the site. 
The database will provide sponsor information, project descriptions, expected results, 
lessons learned, etc. for projects funded under the NEP. 

OEES will also analyze the results of projects completed each fiscal year, based on 
sponsors’ final reports. These syntheses will be posted on the OEES website, 
providing the public with a brief overview of the projects funded through the 
AEEESP, their lessons learned and their findings. 

In response to the lack of systematic dissemination by sponsors, proposals under the 2007 
AEEESP CEP must include an active process for disseminating the results and/or final 
products to the targeted organizations. Posting on a website is not considered to be an 
active method of distribution, unless accompanied by active efforts to inform potential 
organizations about what is available on the website. Partnerships with other 
organizations and/or governments are encouraged in order to assist with disseminating 
results and products. 

In addition, as a condition for multi-year operational contribution funding, P/T 
coalitions and national organizations will provide OEES with a list of organizations to 
whom information, knowledge, tools and best practices will be disseminated. 

Conclusion 

To a great extent, the findings of the interim evaluation are already being incorporated into 
the AEEESP. The findings discussed above provide a target for improvements and contribute 
to the new program’s work to improve accountability and results measurement. 
The evaluation lessons and recommendations will be integrated into the implementation 
evaluation of AEEESP, planned for 2008-2009. 
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1. Introduction and Context 



This section provides a brief overview of the National Literacy Program (NLP) and 
a description of the context for the interim evaluation. 



1.1 Program Description® 

The federal government has had a long-standing involvement in literacy policy and 
program development, driven in large part by the recognition of the impact of literacy 
skills on the inclusion of individuals in society and the economy, and the broader impacts 
of literacy levels on overall societal, economic and labour market performance. 

Since the late 1960s, the federal government has adopted an active and lead role in 
stimulating a modem labour market, in partnership with the provinces. The Department 
of the Secretary of State initially had responsibility for literacy policies and 
programming. This responsibility was eventually transferred to the then Department of 
Manpower and Immigration as part of developing an active employment policy in the late 
1960s. Support for Basic Training for Skills Development and income assistance for 
individual trainees was provided through successive federal skills training programs, 
as part of this active labour market policy. Indeed, in at least some jurisdictions, the 
offering of literacy and academic upgrading programs for adults was largely shaped 
by federal policy and funding levels. 

Beginning in the late 1970s, there was a growing concern about the amount of federal 
funds being expended on academic upgrading, programming that was clearly within 
provincial jurisdiction. This led to a move away from direct federal funding for literacy 
and academic upgrading program delivery. However, a full federal departure from this 
policy area would have left a considerable gap, given the need for new program 
development for the knowledge society, the growing need for research on literacy issues 
in various sectors and settings, and the particular literacy needs of groups facing barriers 
to social inclusion. In particular, the direct linkage of strong literacy skills with individual 
and overall labour market functioning was recognized and gave some urgency to shaping 
an appropriate federal role. 

In this context, the National Literacy Secretariat (NLS) was formed in 1988 to carry out 
an important leadership and “enhancement” role in research and program development 
and capacity building, a role it carried out in partnership with the other national 
organisations, the provinces and community -based organisations. The NLS mission was 
to ensure that Canadians have opportunities to develop and improve the ever-expanding 
literacy skills needed to function at work, at home and in the community, and to support 



^ Sources: Evaluation Terms of Reference; www.hrsdc.gc.ca/en/hip/lld/nls/GC/pdesc.htm ; Treasury Board. Annex "A" 
Terms and Conditions For Grants and Contributions: National Literacy Program. 12 December 2004; The Governance 
Network. National Literacy Secretariat Draft Interim and Summative Evaluation Strategy. Febmary 2005; REP: 
Development of a Comprehensive Evaluation Framework for the Interim and Summative Evaluations of the NLS. 2004. 
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the development and dissemination of information on literacy issues to increase 
awareness and understanding. Its mandate was shaped by the following needs: 

• Innovation: coordination of effort leading to efficiency and effectiveness in hteracy apphed 
research, innovation and sharing of the outcomes of this research - particularly with 
end-users; 

• Access: promoting take-up and access to programs; 

• Inclusiveness: addressing the specific needs of groups, such as Aboriginal people, persons 
with disabilities, families, and those in the workplace; 

• Policy coordination: playing a lead role in the development of a coherent literacy policy 
across federal government departments; and 

• Communications and social marketing: promoting understanding of literacy issues. 

The NLS did not deliver programs directly, but provided financial and technical 
assistance for activities that promoted its goals. The grants and contributions program, 
administered through the National Literacy Program (NLP), was the main instrument 
used to engage key partners in literacy projects. 

The NLP had two key funding streams: 

1. The Federal/Provincial/Territorial (FPT) Stream provided funds for community-based 
organisations based on joint federal/provincial/territorial priorities. Within this 
arrangement, the provinces and territories matched, if not exceeded, NLP spending. 

2. The National Stream provided funds for projects with a national scope. Targeted 
stakeholders included national organisations, provincial coalitions, business and 
labour organisations, and non-literacy organisations, which were viewed as being able 
to bring the literacy community together and act as advocates for the issue while 
providing a forum for non-governmental interests. 

NLP funding was provided for the following categories of eligible activities: 

• The development of applied research in adult literacy (Research); 

• The production of literacy information materials and tools (Learning Materials); 

• Public awareness activities on literacy (Public Awareness); 

• The development of literacy access and outreach measures and project initiatives 
(Access and Outreach); and 

• Coordination and information sharing (Coordination and Information). 

The expected outputs and immediate outcomes of the NLP correspond the above categories 
of eligible activities (see logic model in Appendix E). In addition, a sixth category of outputs 
and immediate outcomes pertains to the development and improvement of strategic literacy 
partnerships. 
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In addition, in 2003-2004, the NLP received increased funding to support the Action Plan 
for Official Languages (APOL) Family Literacy Initiative targeting Francophone 
minority language communities for the development of culturally and linguistically 
suitable family literacy programs; the dissemination of a family literacy project 
inventory; and foundational and models training in family literacy for practitioners. 

The NLP administrative database contained the following grants and contributions budget 
amounts for the years covered by the evaluation: 

. 2002-2003: $27,527,975 

. 2003-2004: $38,351,840 

. 2004-2005: $28,138,162 

On April 1, 2006, Human Resources and Social Development Canada’s (HRSDC) three 
existing adult learning and literacy grants and contributions programs (the National Literacy 
Program, the Office of Learning Technologies and the Learning Initiatives Program) were 
integrated under the Adult Learning, Literacy and Essential Skills Program (ALLESP). 
This integrated program enhances HRSDC’ s capacity to respond to the needs of its key 
partners and communities across Canada in the area of adult learning, literacy and essential 
s ki lls. The integration is expected to result in efficiency and effectiveness gains through 
improved co-ordination of program activities, streamlined procedures and better allocation of 
resources. The current interim evaluation of the NEP was designed prior to the creation 
ofAEEESP. The evaluation methodology was therefore not affected by this change. 
However, results from the current evaluation will no longer lead to a summative evaluation 
of the NEP but will be used to inform a future evaluation of AEEESP. 



1.2 Evaluation Context 

This report presents the results of the interim evaluation of the National Eiteracy Program 
conducted between July 2005 and December 2006. 

As part of Treasury Board’s Policy on Transfer Payments, departments are required to 
produce Results-Based Management and Accountability Erameworks (RMAE) for their 
grants and contributions programs. The frameworks include provisions for periodic 
evaluations of these programs, including interim evaluations that aim to assess the programs’ 
activities and immediate outcomes. 

The purpose of this interim evaluation was to acquire information to help strengthen 
or improve the overall performance and outcomes of the NEP. The list below presents the 
evaluation issues retained to address this evaluation objective. A full evaluation matrix is 
included in Appendix A. 



’ Although the initial Terms of Reference for this evaluation referred to an Interim Evaluation of the NLS, this interim 
evaluation was intended to focus solely on the NLP. The title of the evaluation and the evaluation issues and questions 
were therefore adjusted accordingly. 
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Rationale and Relevance 



RATIONALE AND RELEVANCE 
Rationale and Relevance 

1 . Are the objectives, activities, outputs and desired outcomes of the NLP consistent 
with departmentai, pubiic poiicy, iiteracy community, and public needs and priorities? 

2. Is there a legitimate and necessary role for the federal government in this program area 
or activity? Is the program a candidate for realignment with the provinces? 

IMPLEMENTATION 

Program Utilization and Reach 

3. To what extent do NLP interventions reach targeted stakeholders and learners? 

Program Implementation, Efficiency and Effectiveness 

4. Are eligibility criteria clear and well understood? 

5. Is the current approach to determining priorities and distributing funding the most efficient 
and effective way for NLP to achieve its objective? 

Performance Measurement 

6. Are performance measurement systems and monitoring strategies adequate to support 
the NLP in achieving its objective? 

7. Is performance information being used for decision-making purposes on an ongoing basis? 

SUCCESS 

Achievement of NLP Objectives 

8. Has the NLP made progress toward achieving its objective in each of its targeted 
activity areas? 

Initial Learner Impacts 

9. Does the NLP contribute to the achievement of stronger literacy skills or other impacts 
among learners? 

Initial Capacity Building 

10. Does the NLP contribute to increased capacity among stakeholders? 



Furthermore, the NLP received increased funding in 2003-2004 to support the Action Plan 
for Official Languages (APOL) Family Literacy Initiative. The following question was 
examined to assess the extent to which these activities were being implemented as intended: 

• To what degree can HRSDC report on the literacy models and programs targeting 
families in Official Language Minority Communities? 

In addition, the interim evaluation was intended to collect baseline information to feed into 
a summative evaluation and ensure that reporting mechanisms were in place to collect the 
data required in the summative evaluation. While this objective is no longer valid since 
the program no longer exists, the following questions were examined: 
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• Determine whether sufficient data was being collected to inform the summative evaluation 
and opportunities for improvement to fill any potential gaps; 

• Assess whether the changes brought to the NLS logic model following the 2002 
evaluation, including the clarification of the program definitions, objectives, and targets, 

o 

made achievement of NLP objectives more measurable; and 

• Assess whether new data collection requirements and methodologies have been put 
into place since the 2002 evaluation to ensure that both short and long-term outcomes, 
including changes made since the 2002 summative evaluation, can be measured. 

Finally, the interim evaluation assessed NLP performance in relation to commitments 
made in the NLS Management Response to the 2002 summative evaluation, including: 

• Increasing the transparency of funding decisions; 

• Increasing the accessibility of projects and project results; 

• Simplifying the funding application process; 

• Increasing the reliability and credibility of local research projects; 

• Increasing the usage of contributions; and 

• Developing a framework for a Pan-Canadian Literacy Strategy. 



The logic model was to be revised to reflect the NLS’ foundation as a partnership based model and to more clearly 
identify that the NLS is one of many players contributing to improving literacy skills of Canadians. 
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2. Methodology 



This section provides an overview of the methods developed and implemented for the 
evaluation (scope 2002-2005), and a description of the li mitations associated with the 
methods. 



2.1 Evaluation Methods 

The approach used to evaluate the National Literacy Program (NLP) is based on multiple 
lines of evidence. That is, more than one method was used to measure each of the 
evaluation indicators, thereby strengthening the validity of the findings. The following 
picture illustrates the various qualitative and quantitative methods used. 



Figure 1 

Evaluation Methods 
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Key Informant 
Interviews NLP 
representatives 
(n=7) 



Surveys 

- Combined web/phone survey 
of funding recipients (n=191) 

- Online survey of end-users (n=69) 

Case Studies 

- Project case studies (n=20) 

- Organisation case studies (n=6) 

- Provincial/Territorial case study (n=1) 

Expert Panel 

Expert assessment of 1 1 NLP-funded 
products by eight literacy experts 



Interviews with Unfunded Applicants 

Follow-up interviews with unfunded 
organisation (n=2) 



Furthermore, an iterative approach was taken to integrate the methodologies so that each 
data collection activity would build upon the others. For instance, project case study 
participants were selected from project survey respondents so as to use their survey 
responses regarding availability of end-user^ contact information as a case selection 
criterion. This was to enable the use of case study interviews to obtain end-user contact 
information for the survey of end-users, and the use of end-user survey results to validate 
case study results. Similarly, the NLP-funded products examined by the expert panel 
were selected from the project case studies, this approach offering the possibility of 
providing experts with the summary case study reports containing background 
information on the products. A brief overview of each method used is provided below. 



^ For the purpose of this evaluation, end-users were defined as the immediate intended beneficiaries of the NLP-funded 
activities. 
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2. 1. 1 Administrative data review 



GMAX is a proprietary software package acquired by NLP to assist in the management of its 
grants and contributions data. Program data extracted from the GMAX database for the fiscal 
years 1997/98 to 2004-2005 were analysed to examine changes in the types of projects or 
investments made by NLP over the years, including changes in the number of projects funded, 
the level of funding, the target groups and issue areas funded, and the use of contributions 
versus grants. The GMAX database was also examined for its potential to provide ongoing 
performance measurement information, including an assessment of the data quality. Finally, 
accessibility of NLP projects and outputs and the quality of the projects funded were assessed 
by examining changes over time in the number of NLP project products/reports available or 
accessible through the National Adult Literacy Database (NALD); the number of downloads 
of NLP-funded vs. non-NLP-funded products from NALD; and online quahty ratings 
provided by users on the NLP-funded products downloaded from NALD. The original 
approach intended for this component of the evaluation entailed selecting a sample of 
40 projects that were funded during the NLP evaluation period in order to determine if their 
outputs (reports and tools) were posted to NALD and, where possible, the extent to which 
these products were downloaded. This approach was modified to examine all products posted 
on and downloaded from NALD from 1991 to 2004 because this new approach offered 
a more comprehensive coverage than a sample of 40 projects. Results from an available study 
on the quality ratings of a sample of products posted on NALD^° were also used as an 
additional source of data on NLP products’ quality. 



2. 1.2 Preiiminary key informant interviews with NLP 
representatives 

A total of seven (7) preliminary key informant interviews were conducted with NLP 
representatives to gather information on the NLP, its target clientele, and available 
administrative data at the design stage of the evaluation to assist the evaluation team in 
refining and finalizing the evaluation methodology. The key informant interviews were also 
used to gather some evidence on a number of evaluation issues, focusing on relevance 
to government and departmental priorities and on program performance measurement. 
Key informants included all key (representing each funding stream and corporate functions) 
NLP managers. 



2. 1.3 Review of documents and iiterature 

Over 70 program documents, reports and articles were reviewed to develop a profile of the 
program, to inform development of the data collection instruments, and to address a number 
of evaluation issues pertaining to the relevance, implementation and success of the program. 
All available and relevant documents and reports were identified in consultation with the 
Project Authority and Program representatives, and through a review of existing Canadian 



NALD. Evaluation of Educational Ereeware and Shareware Software Programs as Effective Resources for Adult 
Literacy Training Programs, http://www.nald.ca/software/ 
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and international literature and statistics, including the International Adult Literacy S ki lls 
Survey (lALSS) from Statistics Canada. It must be noted that no document or literature was 
found on comparable literacy programs in Canada or internationally. As well, no documents 
were found, with the exception of past evaluations of the NLS, that presented evidence of 
NLP impacts on literacy learners or literacy capacity building. 



2. 1.4 Survey of NLP funding recipients 

An online survey was conducted of all organisations that received NLP funding during 
the fiscal years 2002-2003, 2003-2004 and/or 2004-2005. This survey was implemented 
within the first eight months of the evaluation. It aimed to produce a profile of funded 
organisations and their project end-users and to document funding recipients’ perception 
of the relevance, implementation and success of the NLP. Being a less costly method than 
a phone survey, the online survey allowed evaluators to invite all NLP funding recipients 
for the targeted period rather than the sample of 40 initially intended. 

For organisations with more than one project in a fiscal year or multiple projects across years, 
the largest project was selected as the reference project for the survey. In total 513 funding 
recipients, with a total of 1,058 projects among them, were identified using this methodology. 

Since the GMAX database did not contain e-mail addresses, these were obtained from 
HRSDC’s Common System for Grants and Contributions (CSGC) database. The CSGC 
is a department-wide system for recording grants and contributions information. Since 
this database contained e-mail addresses for a limited number of organisations, additional 
e-mail addresses were obtained through an Internet search. Out of the population of 513 
organisations extracted for the survey, e-mail addresses were located for 402 
organisations. All 402 organisations were sent an e-mail to invite them to participate in 
the survey. Reminder e-mails were sent to non-respondents and a follow-up telephone 
call was made to non-respondents to ensure the e-mails were sent to the appropriate 
person and to update the e-mail information if necessary. According to the respondent’s 
preferences, five interviews were conducted by telephone during the follow-up contacts. 

A total of 191^' of the 402 funding recipients invited to participate in the survey completed 
the questionnaire, a response rate of 48% (37% of the entire population). The sampling error 
for this sample is approximately 5.6%, based on 191 respondents and a population of 513. 
A comparison of the administrative data for the original 513 projects and the same data for 
the 191 funding recipients who responded to the survey showed no large systematic response 
bias (less than 5 percentage points difference) in terms of the observable variables (Region, 
Funding Stream, and Activity Field). 
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2. 1.5 Project case studies 

Case studies were conducted on twenty (20) projects funded by the NLP between 2002 and 
2005 to analyze evaluation issues in regard to program’s relevance, implementation and 
success. Case studies were selected based, among other criteria, on the survey of funding 
recipients. All projects for which the survey respondents said they kept an electronic list of 
their end-users and might be wilhng to share it for the purpose of a survey of end-users, were 
selected. This selection approach was chosen so as to maximize evaluation resources by 
using the case studies as a means of collecting end-user contact information. The need for 
this approach was dictated by the absence of a database containing contact information on 
NLP project end-users. Fourteen funding recipients said they kept an electronic list of their 
end-users and might be willing to share it for the purpose of a survey of end-users. Since the 
proposed methodology entailed twenty case studies, additional case-study participants were 
selected among project-funding recipients who had conducted an internal evaluation of their 
project (expecting that these evaluations would provide some evidence of end-user impacts 
that could be integrated in the case study). The resulting list of cases covers all five areas of 
activity of the NLP, all regions of the country, as well as all of the NLP’s various funding 
streams, including national and provincial/territorial grants and contributions. Please refer to 
Appendix B for a list of selected cases. 

For each case study, relevant project documents were reviewed and three to five interviews 
were conducted by phone with a combination of project representatives, project partners and 
project end-users (either practitioners or learners), as deemed appropriate for each individual 
case. A total of 26 project representatives, 20 project partners, 30 practitioner end-users, and 
4 learner end-users were consulted as part of project case studies. The results of the survey of 
funding recipients were also used as an additional source of information on the project. 



2. 1.6 Organisation case studies 

Six organisation case studies were conducted to address most evaluation issues as well as 
to examine how three different types of NLP support have impacted the organisations’ 
ability to contribute to the achievement of NLP objectives and to determine whether or 
not the investment has been incremental and results sustainable, particularly at the learner 
level. The case studies included organisations that have received NLP funding 
consistently over at least three years between 2002 and 2005, either as part of one large 
project (2 cases) or a succession of one-year projects (2 cases), and organisations that 
received one-time funding for a period of 12 months or less (2 cases). 

The selection of cases was based on the following criteria: 1) the organisation did not 
take part in a project case study (to avoid over- solicitation); 2) coverage of the NLP’s 
five funding activity areas (this criteria could not be met due to the limited number of 
organisations that were funded in some of the activity areas); 3) coverage of various type 
of organisations (national, provincial, community based); 4) coverage of various project 
sizes in terms of funding; and 5) geographical distribution. The list of organisations 
selected for case studies is presented in Appendix C. 
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For each case study, relevant project documents were reviewed and six to twelve interviews 
were conducted with a combination of organisation representatives and practitioners and 
learners who benefited from the organisation’s work, as deemed appropriate for each 
individual case. A total of 18 project representatives, 19 practitioner end-users, and 1 1 learner 
end-users were interviewed as part of organisation case studies. 



2.1.7 Pro vince/territory case study 

One case study was conducted to assess the relevance, implementation and success of the 
NLP at the provincial/territorial level. Prince Edward Island (PEI) was selected for this case 
study as this relatively small jurisdiction permitted substantial coverage of the key provincial 
organisations involved in literacy and the NEP. Although likely not representative of 
other Canadian provinces due to its small size, PEI offers an example of a federal- 
provincial partnership, which is a key feature of the NEP delivery model. The sources 
used for the case study included interviews with an NEP (1 interview), PEI provincial 
government (2 interviews) and funded organisations’ representatives (11 interviews); 
a review of project and provincial documents; and site visits to three projects. 



2. 1.8 Survey of NLP project end-users 

An online survey of NEP projects end-users was conducted to assess the quality and 
usefulness of the materials produced as part of NEP-funded projects; how they were used and 
applied; as well as their overall impact on end-users. “End-users” were defined as the literacy 
practitioners who used the materials and/or tools produced by NEP-funded organisations. 
(The survey of funding recipients and project case study interviews revealed that for a great 
majority of NEP funded projects the immediate end-users were literacy practitioners). 

Contact lists of project end-users were requested from funding recipient representatives who 
took part in a project case study interview. A total of seven organisations submitted a list of 
end-users in response to this request. Seven other organisations had initially agreed, as part 
of the survey of funding recipients, to submit a list of end-users but later changed their minds 
due to concerns about client confidentiality. The seven lists obtained contained a total of 
213 end-user contacts. This list of contacts was further reduced to 177 names because 36 
e-mail addresses were invalid (the e-mail message bounced back). In order to increase the 
potential number of end-user respondents to this survey, a request was also sent to all 
participants in the survey of funding recipients who said they had a contact list of their end 
users but were unwilling to share it, asking them to distribute the survey invitation to their 
end-users on behalf of the evaluation team. Three organisations distributed the survey 
invitation to their end-users (the total number of end-users who received the invitation is 
unknown). Upon survey closing, 40 questionnaires had been completed by individuals in the 
initial list of 177 names (22.6% response rate) and 29 were completed by individuals who 
had received the survey invitation from one of the three funded organisations, for a total of 
69 completed questionnaires. The response rate for this second means of distribution is 
unknown because the survey invitation was distributed by the funded organisations to an 
unknown number of end-users. 
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2.1.9 Expert Panel 

Eight literacy experts were recruited to review the quality of a sample of NLP-funded 
products. The purpose of the expert panel was to establish a set of criteria to be used in 
assessing research reports, to determine the quality of the research and materials and tools 
developed with the assistance of NLP funding, and to assess the extent to which the NLP 
is meeting the research and materials and tools development needs of the Canadian 
literacy community. Experts were chosen from across the literacy community to include 
a combination of researchers, materials developers, program administrators, and literacy 
trainers. Potential candidates for this panel were identified from a search of the Internet 
and in consultation with NLP representatives. In the interest of objectivity, efforts were 
made to ensure that panellists were not recent recipients of NLP funds or members of 
organisations in receipt of NLP funding, and that they had no involvement in developing 
the materials being reviewed by this expert panel. 

A sample of eleven products was selected containing training tools/materials as well 
as research reports. The initial intended approach consisted of selecting products from 
projects that were reviewed as part of project case studies. The purpose of this selection 
approach was to take advantage of the background information collected as part of case 
studies on the projects from which the products were selected. However, products were 
identified and obtained for only five of the twenty project case studies. The remaining six 
products to be reviewed as part of this expert panel were selected from HRSDC’s library 
of available products produced as part of projects funded by NLP between 2002 and 
2006. This selection entailed picking every eight product among a total of forty products 
filed in HRSDC’s library. The resulting selection yielded a reasonable number of 
research reports (3), thereby removing the need to conduct purposive sampling. 

The experts were asked to each review two or three NLP-funded products, depending on the 
length of the materials to review. Each product was reviewed by two experts. They were 
asked to assess and provide qualitative descriptions of the material on the following 
dimensions: 1) overall quality of the product; 2) relevance to the goals of the project for 
which the product was produced; and 3) value of the product to the field of literacy. 
Two rounds of assessment were conducted, where experts were given a chance to review 
their comments based on the other expert’s assessment. The list of products reviewed is 
included in Appendix D. 



2. 1. 10 Phone Interviews with Unfunded Organisations 

Phone interviews were conducted with representatives of organisations that were 
unsuccessful in obtaining NLP funding for one of their literacy projects in the fiscal years 
2002-2003, 2003-2004 and 2004-2005 (fiscal years corresponding to the scope of the 
evaluation and identical to those used for the survey of funding recipients). The interviews 
focused on the relevance of the program, the adequacy of its funding criteria, and the 
effectiveness of its application and review process. These interviews were also conducted to 
find out if these organisations went ahead with their project in spite of being refused NLP 
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funding and, if they went ahead, find out how the absence of NLP funding may have affected 
their project in terms of scope, timelines, activities, partners and impacts, among others. 

Criteria used for selecting unfunded organisations were: 1) organisations that submitted 
an unsuccessful funding application during the three fiscal years examined; and 2) that 
did not receive NLP funding at all (e.g. through other funding applications) during the 
same period of time examined. A total of 36 proposed projects were not funded by the 
NLP from 2002 to 2005. These 36 projects had been submitted by a total of 30 different 
organisations. Of these 30 organisations, all except three (3) had received NLP funding 
for at least one project during the period examined and were therefore removed from the 
list of unfunded organisations eligible for interviews, thereby leaving three unfunded 
organisations to be contacted for an interview. Two of the three organisations accepted to 
be interviewed (they had not gone ahead with their project without NLP support); the 
third organisation refused due to time constraints. 

This component of the evaluation was not in the initial methodology work plan. It was added 
at the end of the data collection process to complement existing lines of evidence with 
feedback from stakeholders who did not benefit from the program. This added line of 
evidence was suggested by peer reviewers to ensure a more balanced view of the program, 
given that other lines of evidence (the panel of experts excepted) involved individuals 
(program representatives, funding recipients, project partners and project end-users) who 
either have a stake in the program or have benefited from it. 



2.2 Limitations 

As with any evaluation study, there are a number of limitations associated with this 
evaluation. The reader is encouraged to take the following limitations into account when 
reviewing the findings from this report: 

Measuring learner impacts: The NLP is one stakeholder among many who attempt to 
impact literacy levels at the federal, provincial and community levels. It does not fund 
direct delivery of literacy services to learners but, rather, plays a facilitator role and 
provides complementary funding to organisations. Assessing learner impacts attributable 
to the NLP alone is therefore inherently difficult. 

Furthermore, no consensus exists within the Canadian literacy community as to appropriate 
measures of the impact of literacy services on the literacy levels of Canadians. While the 
International Adult Literacy Survey (lALS) and Adult Literacy and Life Skills (ALL) Survey 
(from Statistics Canada) results are generally recognized as good indicators of the need for 
hteracy supports in Canada, these tools are not appropriate measures of the impact of the 
NLP on literacy levels in Canada. 



The NLS logic model states that its ultimate outcome is to “[c]ontribute to stronger literacy skills among Canadians 
who participated in NLS funded projects and initiatives.” 
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Since the majority of NLP funding (at least three quarters) was provided through grants 
(which entail minimal accountability requirements), there were limited requirements for 
the recipients to collect and report information on their specific end-users. The resulting 
absence of a database of NLP project end-users limited the ability of the evaluation team 
to implement a survey of end-users and to measure end-user impacts. 

Finally, evidence collected as part of this evaluation showed that learners make up 
a relatively small proportion of immediate end-users of NLP project outputs, thereby 
increasing the difficulty of obtaining access to learners for the purpose of measuring learner 
impacts. As a result, assessment of learner impacts was limited and based mostly on 
testimonies from funded organisations’ representatives. However, this is to be expected since 
learners are not the immediate targets of NLP activities. 

Overall impact measurement: The absence of program baseline data and specific 
benchmarks other than the project profile information available as part of the NLP 
administrative data has limited the evaluators’ ability to compare current program 
achievements to the period preceding the period targeted by the evaluation. The resulting 
conclusions are therefore based on post-intervention measurement of various program 
stakeholders’ perceptions. 

Case Studies: The practitioners, project partners and learners consulted as part of project 
and organisation case studies were selected and recruited by the funding recipients, 
thereby introducing a potential bias in the testimonies received. This selection approach 
was chosen because, for privacy reasons, the majority of funding recipients refused to 
provide full contact lists of their partners and end-users, preferring to request and arrange 
their participation themselves. Nevertheless, the twenty project case studies document the 
wide range of projects benefiting from the NLP’s contribution while providing lessons 
learned and suggestions for program improvement. 

End-User (practitioners) Survey: The survey of end-users relied on the input from 
end-users of only seven organisations out of 513, representing only 1.4% of all organisations 
that received NLP funds from 2002 to 2005. Consequently, results from this end-user survey 
are not considered to be representative of NLP end-users. The information was therefore used 
for illustrative purposes and no general conclusions about the program were derived from 
these responses. It must however be noted that, in spite of the limited number of end-user 
lists identified through the survey of project funding recipients and obtained through project 
case studies, the selection approach adopted was valuable in that it enabled the evaluation 
team to identify and document the challenges and barriers faced by the NLP in collecting 
end-user contact information for the purpose of an end-user survey. Given that the survey of 
funding recipients invitation was sent to all organisations funded by the NLP within the 
period targeted by the evaluation, and that all organisations who reported having a contact list 
of their project end-users were invited to participate in project case studies and to submit their 
list of end-users, the evaluation team concludes that it would likely not have had more 
success in obtaining end-user contact information through other means (e.g. contacting 
funding recipients who did not answer the survey of funding recipients). 
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Expert Panel: The sample of eleven products reviewed by the expert panel is not 
representative of the wide variety of products resulting from NLP-funded projects. 
Consequently, general conclusions on the quality and relevance of NLP-funded products 
cannot be drawn from these results. 
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3. Key Findings 



Key evaluation findings are presented under each relevant evaluation issue. 

3.1 Rationale and Relevance 



Evaluation Question: Are the objectives, activities, outputs and desired outcomes of the 
NLP consistent with departmental, public policy, literacy community, and public needs 
and priorities ? 

Departmental priorities pertaining to literacy during the time period examined are outlined in 

the Department’s 2002-2003 Report on Plans and Priorities (RPP), which states that 

“[sjupport for literacy activities through the National Literacy Secretariat (NLS) will continue 

to be a fundamental part of our approach to s ki lls and learning. We will continue to promote 

literacy as a key component of a learning society and to make Canada’s social, economic and 

community life more accessible to people with limited literacy skills.” Furthermore, with its 

strategic priority, “Through access to learning, Canadians can participate fully in 

a knowledge-based economy and society”, formulated in its 2004-2005 RPP, the Department 

vowed to “[pjursue policy and program development on lifelong learning, focusing on the 

reduction of financial and non-financial barriers to learning.” Low literacy skills were 

identified as one such barrier. Literacy was also identified as one of six priority areas for 

labour market collaboration with provinces. The National Literacy Program (NLP) is 

consistent with these Departmental priorities given that it works in partnership with the 

provinces, territories, non-govemmental organisations and business and labour organisations 

“to promote literacy as an essential component of a learning society and to make Canada's 

1 ^ 

social, economic and political life more accessible to people with weak literacy s ki lls.” 

Key informants from NLP also considered that the NLP expected outcome of improved 
access to literacy skills by Canadians with low literacy s ki lls is aligned with Human 
Resources and Social Development Canada (HRSDC’s) mandate, “to improve the standard 
of living and quality of life of all Canadians by promoting a highly s ki lled and mobile labour 
force and an efficient and inclusive labour market,”*^ given the link established between 
literacy skills and an individuals’ access to the labour market and social, economic and 
political life in our new knowledge-based economy. 

“The NLP’s goal is to promote economic vitality, productivity, and 
competitiveness through higher literacy skills. Literacy is l ik e Velcro to which 
other s ki lls attach.” (Key Informant from NLP). 

The implementation of a federal partnership literacy funding program such as the NLP 
also appears to be consistent with the Government of Canada’s public policy priorities for 
the time period examined. In the 2002 Speech from the Throne, the Canadian government 
vowed to “build on its investments in research, literacy and education, and in competitive 



http://www.hrsdc.gc.ca/en/gateways/nav/top_nav/program/nls.shtml 
Evaluation Terms of Reference. March 2005. 



Interim Evaluation of the National Literacy Program 





cities and healthy communities.”^^ More specifically, in its response to “Raising Adult 
Literacy Skills: The Need For A Pan-Canadian Response”, the report submitted in 2003 by 
the Standing Committee on Human Resources Development and the Status of Persons with 
Disabilities, the Government of Canada stated that it viewed “literacy issues as having 
priority in the context of an innovation and learning culture” and that “all partners need to 
work together as part of a concerted effort to address literacy issues in Canada.” The response 
clearly identifies the National Literacy Secretariat as one of the mechanisms through which 
the Canadian government is playing an active role in supporting various literacy initiatives 
and the development of partnerships.*^ 

By contributing to “stronger literacy skills among Canadians”, the NLP also addresses 
public needs related to Canadians’ ability to participate in the new knowledge economy. 
The federal government’s Knowledge Matters innovation paper published in 2002 
underscored the need for Canada to strengthen the skills and learning of its people and, 
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“to provide the opportunity for all to contribute to and benefit from the new economy.” 
Among the goals outlined in this paper, the following figured strongly: increased job 
security; more effective job performance; and increased ability of adults to adapt to 
change and participate fully in Canada’s society and economy. 

Results from the survey of funding recipients also confirm the relevance of the program: 77% 
(out of 191) of funding recipient respondents rated (on a four point scale where 1 was not at 
all consistent, 2 was not very consistent, 3 was somewhat consistent and 4 was very 
consistent) the NLP as somewhat or very consistent with literacy needs and priorities. 
When asked what needs and priorities of the literacy community the NLP should help 
address, the needs most frequently mentioned by survey respondents were for longer-term 
or core funding (41 mentions), NLS coordination and networking (29 mentions), awareness 
and promotion activities (23 mentions), more support for research projects (18 mentions), 
funding for direct learner service delivery (18 mentions), and funding for training or 
accreditation of practitioners (15 mentions). 



Key informants consulted as part of one case study recognized the essential role played by 
the NLP due to its leadership, its unique ability to address literacy capacity gaps, and its 
support for innovative concepts such as informal learning, that are not supported by other 
funding organisations. 



On the one hand, long-term funding, funding for core programming (as opposed to 
projects) and funding for direct learner service delivery being excluded from eligible 
NLP-funded activities, the program appears to fall short of these key areas of need 
identified by survey respondents. The survey open-ended responses also revealed some 
gaps with respect to awareness building both with respect to potential participants and the 
Canadian public. Survey respondents referred to the need for funding of awareness and 
promotion activities to publicize programs and services to recruit participants and the 



Speech from the Throne to Open the Second Session of the 37th Parliament of Canada, September 30, 2002. 
http://www.pco-bcp.gc.ca/default.asp?Language=E&Page=sftddt&Doc=sftddt2002_e.htm 

Government of Canada. Response to: “Raising Adult Literacy Skills: The Need For A Pan-Canadian Response” 
The Report of the Standing Committee on Human Resources Development and the Status of Persons with Disabilities. 2003. 
HRSDC, 2002. 
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need for direct awareness building activities to build public support to help reduce the 
stigma associated with low literacy, which prevents many individuals from seeking 
assistance. Organisation case studies also highlighted the need to disseminate research 
findings more widely and in a way that is more accessible and useful to practitioners. 
According to some organisation representatives, much of the current research tends to be 
written up in a very academic format that is not easily accessible to most. 

“It is important to avoid jargon, to use clear language, to get the research 

into the hands of practitioners and learners on the ground, and to develop 

practical tools and resources in response to research findings”. 

{Organisation Representative). 

Key informants from NLP also reported that the need for stable funding continues to be of 
concern for literacy organisations, as confirmed by survey of funding recipients’ responses 
and interviews with unfunded organisations. In an attempt to address this need without 
changing its eligibility criteria that precludes funding of core programming and long-term 
(ongoing) funding, the NLP made the strategic decision in 2003 to fund fewer but larger 
multi-year (2-3 years) projects. Evidence of this shift is apparent from the review of the 
program administrative data, which shows that in the three years covered by the evaluation 
the average number of funded projects decreased by 50% while the average value of the 
projects doubled. It must also be noted that only three of the 513 organisations that submitted 
one or more applications to the NLP between 2002 and 2005 did not receive any funding. 
This finding suggests that while the number of funded projects has decreased, the NLP 
continued supporting almost all the organisations that applied for funding. Consistent with 
the other needs identified by survey respondents, the program has funded activities in the 
areas of coordination and networking ($12,143,017 in 2004-2005 or 43% of total project 
funding), awareness and promotion ($3,067,354 in 2004-2005 or 11% of total project 
funding), and research ($4,771,462 in 2004-2005 or 17% of total project funding). 



The Canadian Automotive Repair & Service (CARS) Council’s Basic Snowmobile 
Maintenance for First Nations project addressed several specific needs: lack of training 
infrastmcture in remote communities, barriers of time and distance to training, and an 
urgent need for practical learner-based training. Several sources consulted for this 
evaluation highlighted the relevance of this type of project given that lack of essential s ki lls 
is a barrier to the acquisition of knowledge and practical s ki lls critical to self-sufficiency 
in remote communities. 



Case studies also provided testimonies from funding recipients attesting to the 
importance of the NLP’s contributions to addressing local literacy needs. Evidence from 
the six organisation case studies shows that NEP funding is being used for projects that 
are deemed highly relevant to address local and regional literacy needs, particularly for 
initiatives targeting specific clienteles such as families, seniors, youth and the workplace. 
NEP funds are seen by organisation representatives as a key enabler of capacity building 
activities in the field of literacy tools development and services, as well as professional 
development and coordination activities. There appears to be few alternative sources of 
funding, in the areas examined, for literacy capacity building, coordination and program 
development. As such, the NEP plays a unique and necessary role. 
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Evaluation Question: Is there a legitimate and necessary role for the federal government 
in this program area or activity? Is the program a candidate for realignment with the 
provinces? 



This evaluation question, stemming from the Government of Canada’s Expenditure Review 
Committee questions, is particularly relevant given the NLP’s intervention in a domain of 
provincial jurisdiction. However, none of the literature reviewed or the NLP key informants 
consulted have presented arguments against the legitimacy and necessity of the NLP. 
The NLP’s partnership role is considered by NLP and PEI representatives to be a legitimate 
and necessary role for the federal government in the area of literacy, given the identified need 
to coordinate literacy activities across provincial and regional boundaries, to foster effective 
partnerships, and to complement and leverage provincial efforts in literacy. 

Notably, the PEI case study has provided a strong example of effective and successful federal 
and provincial collaboration. The provincial government sees the NLP as a valued partner in 
the development and implementation of the PEI literacy strategy. However, provincial case 
study informants felt that there needs to be a strengthened federal mandate and funding for 
literacy development. Informants involved in workplace literacydeaming programs felt that 
the national approach to essential skills development is too separate from literacy. 



Representatives of an NLP-funded organisation in the field of workplace literacy felt 
that NLP support “is extremely important, as few alternative sources of funding exist 
for this kind of work.” According to them, workplace educators have repeatedly 
emphasized the value in having an opportunity to come together for face-to-face 
networking and professional development. Organisation representatives found the NLP 
“a unique and well-suited source of support for this type of activity.” 



“There needs to be stronger recognition of the link between literacy and 
essential skills, and a one-stop approach to funding of programs.” 
{Provincial Case Study Key Informant). 

Several funded organisations and literacy stakeholders consulted as part of organisation 
case studies also emphasized the importance of the role played by the NLP as a champion 
of new approaches and initiatives at the national level and as a source of funding where 
provincial funds are minimal. 



According to representatives of an NLP-funded organisation, the NLP plays a relevant and 
important role because few alternative sources of funding exist to address literacy needs 
in their province. This is particularly tme for family literacy programming. The NLP 
funding is considered to be crucial in advancing literacy efforts in their province. 



According to key informants from the NLP, before the NLP’s creation, in several provinces 
the literacy community was not well coordinated, had an underdeveloped voice, and 
conducted limited advocacy efforts. Since the NLP’s inception, a literacy strategy has been 
developed in every province and territory, and members of the literacy community have 
learned to work with federal and provincial governments. 
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3.2 Implementation 



Evaluation Question: Program Utilization and Reach - To what extent do NLP 
interventions reach targeted stakeholders and learners? 

1 S 

The NLS has listed the following targets for the primary reach of its actions: 

• Provinces and territories; 

• Field- level literacy organisations delivering programs to individuals; 

• Non-govemmental groups whose primary focus may not be literacy;^^ 

• Provincial and national literacy organisations; 

• International organisations; 

• Other federal government departments and agencies; 

• Universities; 

• Researchers; and 

• The media. 

The program’s administrative database does not contain coded information on the types of 
organisations receiving NLP funding, thereby preventing a comparison of actual funded 
organisations to the target categories enumerated above. One general category of profile 
information is however captured in the administrative database, distinguishing between 
literacy, non-literacy and private sector organisations. For the years 2002 to 2005 targeted by 
this evaluation, 292 funding recipient organisations were coded as literacy organisations, 211 
were non-literacy organisations, and 10 were private sector organisations. 

The program’s administrative database also records the end-user groups targeted by 
individual projects at the time of funding application but not the profile or number of 
individuals actually reached. In the absence of precise data on end-users, no definite 
conclusions can therefore be made on the program’s actual reach. 

Nevertheless, some indications of the NLP’s reach could be gathered through the survey 
of funding recipients. First, funding recipients surveyed were asked to provide some 
details on the profile of their organisation. They were typically well-established local 
literacy or non-profit organisations, with small staff and limited annual budgets. Key 
characteristics of the 191 respondents included: 



The Governance Network (2005). National Literacy Secretariat Draft Interim and Summative Evaluation Strategy. 
February 2005. 

These could be, for example, organisations helping homeless Aboriginal people or organisations helping ex-offenders 
reintegrate into society. 

Presumably as disseminators of information on literacy issues in Canada. 
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• Type of organisation - local literacy organisations (35%) or non-profit service organisations 

( 22 %); 

• Years in operation - more than 15 years (62%), 11 to 15 years (19%); 

• Number of employees - 1 to 5 (39%), 6 to 10 (23%); and 

• Annual budget - under 100k (22%), lOOK to 499K (40%). 

The profile of the funding recipients was also analyzed to determine if there were 
differences based on the type of funding or related project administrative data variables. 
The findings from this analysis included: 

• Funding recipients from Quebec (25.7% or 49/191) were more likely to be local 
literacy organisations (62%), as were funding recipients who received funding under 
the Provincial/Territorial funding stream (54%); 

• Local literacy organisations accounted for the majority of the projects approved under 
$24K (65%) and between $25K and 49K (58%); 

• There was a significant decline in the percentage of local literacy organisations 
receiving funding in the first two years of projects covered by the survey (46% in 
2002-2003 and 45% in 2003-2004) and the most recent fiscal year covered in the 
survey (15% in 2005-2006). During the same time period the percentage of provincial 
literacy organisations funded increased from 4% to 8% to 36%; and 

• The percentage of organisations with operating budgets under $100K declined from 
28% in 2002-2003 to 9% in 2005-2006. 

Second, funding recipients surveyed were asked to describe the target groups for their 
NLP-funded products^' and the estimated number of end-users of these funded products. 
It should be noted that this information is based solely on funding recipients’ estimates of the 
intended targets; the relative percentage of the actual end-users of the products could be 
different. 

According to funding recipients surveyed, the primary users or target beneficiaries for 
their principal NLP project outputs were local literacy organisations, learners, other 
NGOs, provincial literacy organisations, and fa mi lies. However, the projects were often 
designed for multiple target groups. 



For the purpose of this report, products include materials, tools, research or services produced as part of projects that 
received NLP funding. 
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Table 1 

Type of End-Users for NLP-Funded Projects 
(Survey of Funding Recipients) 


Primary users or target beneficiaries for the principai 
project outputs 


Percent 

of Respondents 


Local literacy organisation 


65.3% 


Learners 


64.2% 


Other non-profit service organisation 


47.7% 


Provincial literacy organisation 


43.2% 


Families 


42.6% 


National literacy organisation 


24.4% 


University/College/CEGEP 


20.5% 


Other non-profit research/advocacy organisation 


17.6% 


Provincial government 


17.6% 


Employers 


17.0% 


Union/Labour organisation 


13.1% 


Local government 


3.4% 


Other 


19.3% 


Number of respondents 


176 



Approximately two thirds (64% out of 191) of the survey respondents were able to provide 
information on the number of end-users reached by their primary project product. As shown 
in Table 2, a majority of surveyed funding recipients (53% out of 122) who were able 
to answer this question reported that their primary project product was used by 16 to 50 or 5 1 
to 125 end-users. 



Table 2 

Number of Primary Product End-Users 
(Survey of Funding Recipients) 


Number of persons who requested or used the primary 
project product 


Percent 

of Respondents 


1-15 


5.4% 


16-50 


21 .7% 


51-125 


31 .5% 


126-500 


17.4% 


More than 500 


23.9% 


Total 


100.0% 


Number of respondents 


122 



In PEI, case study results show that Literacy/Adult Basic Education programming enrolment 
has increased from zero to 1,200 annually since the development of the NEP-supported 1996 
provincial literacy strategy. Key informants indicated that all major provincial and 
community-based organisations in the province are reached by NEP funding and the 
provincial strategy. 
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Lastly, in four of the six organisation case studies, the project delivered training to several 
trainers who in turn used the new skills acquired to assist literacy learners. The end-users 
thus reached are very diverse, including literacy practitioners, youth, families, and seniors 
from various Canadian provinces. 

In conclusion, although the data collected does not allow definitive conclusions to be drawn 
on the reach of NLP-funded projects, there is some indications that the funded activities are 
targeting and reaching a wide range of stakeholders particularly literacy organisations, 
non-profit service organisations and, indirectly through funded project outputs, literacy 
practitioners and learners, including youth, families, and seniors from various Canadian 
provinces. The program has taken a responsive approach to addressing local literacy needs 
and, as such, did not set specific targets for the reach of its activities. Furthermore, no data is 
being collected by the program on the actual end-users of NLP-funded activities, thereby 
preventing an accurate assessment of its reach. 

Target categories of stakeholders whose reach could not be assessed as part of this evaluation 
include international organisations, other federal government departments and agencies, and 
the media. These categories of potential end-users were not included as response options 
in the survey of funding recipients because they appeared from the review of program 
documents and key informant interviews with NLP representatives to be secondary 
(as opposed to primary) end-users of the NLP. These categories of stakeholders appear to be 
more immediate targets of the work of the NLS than of the NLP, which is only one 
component of the NLS’s activities. 



Evaluation Question: Program Implementation, Efficiency and Effectiveness - Are 
eligibility criteria clear and well understood? 



NLP eligibility criteria are listed in the program application package and on the NLP 

Website. Eligible organisations include: 

• Non-profit organisations operating at the national, provincial, regional, community or 
neighbourhood level; 

• Non-govemmental institutions such as teachers’ federations, labour unions, trade 
associations and professional associations; 

• Canadian post- secondary institutions such as universities, colleges and vocational and 
technical institutes; 

• Provinces and territories; and 

• Provincial/Territorial institutions, including provincial Crown corporations. 
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Eligible types of projects are those that fall under the program’s five categories of activities: 
Learning Materials, Access and Outreach, Coordination and Information, Public Awareness, 
and Research. The NLP does not provide project funding for the following: 

• Provision of direct, ongoing literacy services; 

• Activities that have already taken place; 

• Ongoing activities; 

• Projects which need 100% funding; 

• Capital costs; 

• Travel outside Canada; and 
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• Debts or financial losses that result from a project. 

In addition, the NLP requires that applicants secure additional funding for their project from 
other sources than the NLP. 

According to key informants from NLP, program eligibility criteria are generally well 
understood by project funding applicants and when potential applicants have doubts 
regarding eligibility criteria, they can access the NLP website and have easy phone access 
to program officers. 

A majority of funding recipients consulted as part of the survey and case studies also 
reported that they considered the eligibility criteria as clear. Eighty-three percent (out of 
191) of funding recipient survey respondents indicated that they were satisfied or very 
satisfied with the clarity of the eligibility criteria. This is the highest satisfaction rating 
reported by funding recipients of all the satisfaction questions asked in the survey. 

However, even though 70% (out of 191) of the funding recipients report being satisfied 
or very satisfied with the simplicity of the funding application process, 14% remain 
dissatisfied or very dissatisfied. 



Evaluation Question: Program Implementation, Efficiency and Effectiveness - Is the 
current approach to determining priorities and distributing funding the most efficient 
and effective way for NLP to achieve its objectives? 



Approach to determining funding priorities 

Within the five categories of eligible NLP activities (Learning Materials; Access and 
Outreach; Coordination and Information; Awareness Raising; and Research), specific 
NLP funding priorities for the provincial/territorial stream are established annually 
by each province in discussion with NLP managers and based on provincial-specific 
needs and priorities. National funding stream priorities are established by NLP managers, 
in consultation with stakeholders. According to the NLP managers, in recent years 



National Literacy Secretariat. Grant Funding Application Package. 
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national stream funding priorities have focussed on workplace literacy and family 
literacy, emerging areas deemed important by the program. 

The process for distributing federal/provincial stream funding is considered by NLP 
representatives consulted to be the best approach given the absence of federal jurisdiction 
in areas pertaining to education. They all consider that this collaborative approach to 
priority setting is an effective way to leverage provincial/territorial commitments 
to literacy. Furthermore, the flexibility allowed by the national stream funding allocation 
mechanism is seen by NLP representatives as a key feature enabling the program to 
support the NLS’s national leadership role in the field of literacy. 

A majority of participants (funding recipients) in the organisation case studies reported 
not being aware of the way NLP priorities are determined and funding distributed. In the 
same way, only 57% (out of 191) of funding recipient survey respondents expressed 
satisfaction with the transparency of the proposal review process and 54% with the 
timeliness of the funding approval process. 

Linkage of funding decisions with immediate outcomes 

As shown in Table 3, the NLP’s administrative database reveals that projects funded during 
the period 2002-2005 targeted each of the first five intended immediate outcomes of the 
NLP. While the NLP administrative database does not include strategic partnerships as 
a specific activity category, key informants from NLP and anecdotal evidence collected as 
part of project and organisational case studies indicate that, in all cases, strategic partnerships 
were created for leveraging additional support to NLP-funded projects. It must be noted that 
this was also a program funding requirement. 



Table 3 

Value of Approved Projects by Activity Code and Fiscal Year 


Activity Code 


2002-2003 


2003-2004 


2004-2005 


Coordination & Information 


33.2% 


37.3% 


43.2% 


Learning Materiais 


26.9% 


18.0% 


17.9% 


Pubiic Awareness 


15.5% 


15.2% 


10.9% 


Access & Outreach 


13.5% 


20.6% 


1 1 .0% 


Research 


10.9% 


8.9% 


17.0% 


Totai 


100.0% 


100.0% 


100.0% 



Linkage of funding decisions with target groups 

As discussed above under Program Utilization and Reach, no program administrative 
data was collected on the target groups or on the target end-users actually reached by the 
funded projects. Furthermore, the NLP has taken a responsive approach to establishing 
federal-provincial stream funding priorities at the provincial level; no evidence was found 
that the program has oriented its funding decisions towards specific target groups. 
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Stakeholders are the provincial governments involved in the joint (with NLP) establishment 
of funding allocation priorities and joint selection of winning applications for their province. 
The only parameters imposed by NLP were that projects had to fit within the five areas 
of activity and the fact that no funding would be given for core operations or ongoing 
activities. 

NLS ability to deliver the NLP within budget and meet accountability requirements 

The financial information obtained from the program indicates that the NLP is being 
delivered within budget, that is, the program has not exceeded the budget allocated for its 
implementation. The NLP’s ability to meet TBS and HRSDC accountability requirements 
is discussed in the following section on performance measurement. 



Evaluation Question: Performance Measurement - Are performance measurement systems 
and monitoring strategies adequate to support the NLP in achieving its objectives? 



Performance Reporting Requirements 

Performance reporting requirements for NLP grants and contributions are largely limited 
to final reports (except in the case of interim reports for larger projects funded for more 
than one year). In their proposals, applicants are asked to provide detailed evaluation 
methods for determining the success of their proposed projects. Evidence was found of 
some performance data being collected by funded organisations, mostly in the form 
of participation/attendance statistics and satisfaction/learning assessment feedback 
questionnaires administered at the end of training, workshops and conferences. Most 
completed projects examined as part of case studies and 74% (out of 191) of surveyed 
funding recipients have also implemented an end-of-project evaluation that included 
some form of consultation of their end-users on the outcomes of the project. These results 
are reported in the projects’ final reports. However, the information contained in these 
final reports is not rolled-up or used by the NLP for purposes other than approving final 
project payment (where applicable) and closing the project file. 

According to NLP managers, funding recipients generally understand and comply with the 
reporting requirements. In turn, funding recipients consulted as part of case studies generally 
reported being comfortable with project performance reporting requirements. However, 
a majority reported that they had difficulties monitoring and tracking end-users, particularly 
learners, because they cannot impose rigid rules requiring that learners report on results of the 
funded interventions without potentially impacting the fragile trust relationship created with 
this particularly vulnerable clientele. 

Effectiveness of Management Systems for Grants and Contributions 

According to key informants from NLP, three processes are in place to ensure achievement 
of program objectives and quality assurance of outputs. First, diligent project development, 
assessment, and selection processes at the front-end of each agreement are used by NLP 
officers to provide assurance on the quality of the initiative and demonstrate if the applicant 
has the capacity to implement and complete the project as proposed. Second, NLP officers 
conduct informal monitoring exercises through ongoing client liaison. Third, funded projects 
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are required to submit reports on the progress and achievement of their planned activities. 
The above key informant testimonies are consistent with documented descriptions of NLP 
management activities and findings from project and organisation case studies. 

Financial reporting requirements, however, are more stringent for contributions (e.g. they 
require monthly accounting of expenses). One funding recipient reported that these additional 
requirements exceed their capacity because they do not have a professional accountant on 
staff. Program representatives also expressed concerns that many of the funded organisations 
the NLP funds might have limited capacity to meet contribution reporting requirements. 
They also argued that the type of contribution agreements used for the NLP is an inadequate 
response to the need to increase project accountability because it is limited to financial 
reporting as opposed to reporting on results. 

Appropriateness of Performance Indicators and Targets and Application of Performance 
Information to Improve Internal Systems 

An adequate performance measurement system should generally comprise the following: 
1) clearly defined indicators of the program’s performance (achievement of objectives 
and intended results) with identified sources of information on these indicators; 2) clearly 
identified responsibilities for collecting performance information; 3) standardized tools 
for collecting the performance information; 4) a systematic process for collecting and 
collating the performance information; and 5) an information management system or 
database to collect and store the performance information. 

Results from the document review and key informant interviews indicate that the NLP does 
not have an adequate performance measurement system in place. This conclusion is based on 
the following observations: 1) no performance indicators or targets were officially approved 
by the Program against which the actual performance of the NLP could have been assessed; 
and 2) the Program does not have a system in place to record and store performance 
information, a necessary condition for NLP managers being able to use it for decision- 
making or reporting purposes. It must however be noted that initial attempts were made 
by the NLP to develop performance measures (draft measures are contained in the draft NLS 
Interim and Summative Evaluation Strategy dated February 2005) and to test performance 
data collection tools with a sample of funded organisations. The decision not to follow-up on 
these initial attempts was attributed in part to Senior Management’s desire to focus on the 
new ALLESP, for which a performance measurement system is being developed. 



Evaluation Question: Performance Measurement - Is performance information being 
used for decision-making purposes on an ongoing basis? 



The information collected through NEP project reports and monitoring visits is not integrated 
into any kind of formal analysis or report on the overall performance of the NEP. There is no 
evidence of ongoing use of performance information for decision-making the organisation. 
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3.3 Success 



Evaluation Question: Achievement of NLP Objectives - Has the NLP made progress 
toward achieving its objectives in each of its targeted activity areas? 



The NLP’s objectives correspond to the following expected outputs and outcomes for each 
of its priority activity areas: 

• Learning Materials: The production of literacy information materials and tools 
(output) and Increased availability of adult literacy materials and tools by learners and 
practitioners (immediate outcome); 

• Access and Outreach: The development of literacy access and outreach measures and 
project initiatives (output) and Improved access to literacy skills by Canadians with 
low literacy skills (immediate outcome); 

• Coordination and Information: The production of literacy information materials and 
tools (outputs) and Improved dissemination and sharing of information among literacy 
stakeholders (immediate outcome); 

• Public Awareness: Public awareness activities on literacy (output) and Greater awareness 
of literacy programs, services and issues among targeted sectors (immediate outcome); 

• Research: The development of applied research in adult literacy (output) and Enhanced 
literacy research in identified priority areas (immediate outcome); and 

• Partnerships: The development of partnerships, projects and initiatives (output) and 
Improved strategic partnerships involved in literacy issues (immediate outcome). 

Intended indicators for measuring achievement of the above objectives included the project 
outputs produced; the quality, usefulness, and utilization of tools and materials; accessibility 
and dissemination of tools and materials; best practices being exhibited in the field; public 
awareness and learner participation; creation of partnerships and leveraging of additional 
funding and support. 

Project Outputs 

The NLP database does not contain information on the profile or number of actual project 
outputs. Key project outputs most frequently mentioned by respondents to the survey of 
funding recipients are literacy tools and materials (mentioned by 63% of respondents out 
of 81), partnerships with other organisations (mentioned by 62% of respondents), and 
awareness raising and promotional events and products (mentioned by 60% of respondents). 

Quality, Usefulness, and Utilization of Tools and Materials 

The tools and materials produced in the context of the projects outlined in the case studies 
were typically described by practitioner end-users as highly useful, well researched, 
culturally relevant, and of exemplary quality. 
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“These are simple, not flashy materials that facilitate communication 
between two adults, supporting each other in making the child successful 
in kindergarten.” (Practitioner end-user). 

Expert panellists provided positive assessments of the quality and usefulness of a majority of 
the eleven products reviewed (these products included eight tools - training manuals, a simple 
language document, handout materials, an assessment tool, bulletins, and a website - and 
three research reports). All but two tools were deemed high quality by both reviewers, while 
the three research reports obtained mixed reviews. The qualities attributed to the products 
included, among others, good format/presentation and high potential usefulness to the 
literacy community. Key criticisms pertained to the lack of methodological rigour in the 
development and content of the products. This criticism applied to two tools and all three 
research reports, although negative assessments of the research reports were not unanimous 
among expert reviewers. 



Praise was expressed by practitioners for the quality and utility of the materials and 
workshops developed through one of the NLP-fimded projects. Practitioners remarked that 
the resources were culturally appropriate and effective in meeting the needs of learners. 
The materials were deemed particularly helpful to practitioners in communicating with 
learners of all age ranges. 



The 2002 evaluation had identified limitations in the rehability and credibihty of local 
research. An NLS-commissioned research reviewing NLP-fiinded research reports from 1998 
to 2003 concluded that the NLS had contributed substantially to literacy education in Canada 
through its research function but that improvements could still be made, among others, 
in detailing the research methodology and drawing linkages to other research work. Similar 
concerns were expressed by some experts with regards to a sample of research reports 
examined. No evidence was found that the NLP’s approach to funding research projects was 
modified following this study or to address the 2002 evaluation recommendations. 

Also, 75% (out of 191) of funding recipients surveyed indicated that NLP had contributed 
somewhat or a lot to improving the quality of literacy research, materials and/or tools. 
Sixty-four percent (out of 191) said that the NLP contributed somewhat or a lot to 
improved literacy programming. 

Lurthermore, a review of NLP funded products posted on NALD demonstrated general 
appreciation for the quality of the products. NALD is one of the key partners in the NLP’s 
approach to disseminating the results of its funded projects. Quality ratings of NALD 
products were provided by literacy coordinators and instructors of the Community Academic 
Services Program (CASP) in New Brunswick as part of NALD’s Software Evaluation 
Project. The ratings were based on a 5-point scale (1 being the lowest and 5 the highest 
quality rating). In total, there were 136 products published by NALD between 1990 and 2004 
with these ratings, 44 being NLP-funded. None of the NLP products received a rating below 
three; 43% had a rating of four out of five and 18% had a rating of five. The average rating 
for the NLP products was 3.8; however, there was no statistically significant difference in the 
ratings of NLP and non-NLP-funded products. 



30 



Interim Evaluation of the National Literacy Program 





Practitioners consulted on one NLP-funded project considered the regional workshops 
it held to be positive and highly productive. In these workshops, they were introduced 
to a new model for integrating literacy in their practice and they forged new partnerships 
which led to relevant and practical action plans. 



End-user survey results show that users of the NLP-funded products (respondents to the 
end-user survey) include, among others, the following categories of individuals: 

• Literacy program/services administrators; 

• Literacy trainers/teachers/facilitators; 

• Researchers; 

• Distributors/hosts of literacy materials/curricula/programs/information; 

• Developers/testers of literacy training and education tools and materials; 

• Policy makers or advisors; 

• Literacy volunteers; 

• Employment/training/career/human resources counsellors; 

• Lundraisers and lobbyists; and 

• Students currently studying/training to become literacy practitioners. 

A majority (77% out of 69) of end-users consulted reported that they were still using or 
accessing the NLP funded products and services. End-users were also queried on how NLP 
funded products and services were applied in their literacy-related work. Sixty (60) 
respondents provided multiple open-ended answers that were sorted into the following four 
main categories: 

• Used to improve or support their current literacy training or policy practice (32); 

• Used to develop new literacy programs or training tools (17); 

• Used to create or strengthen partnerships (9); 

• Used to conduct awareness raising activities (7); 

• Distributed the products/disseminated the new knowledge among their network of 
end-users (6); and 

• Used the information to develop stronger funding applications (2). 

In summary, the above results from end-user, funding recipient surveys, expert panel and 
case studies indicate that most project outputs are perceived as being adequate and useful 
by practitioners and stakeholders. 
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Accessibility and Dissemination of Tools and Materials 

The following evidence indicates that NLP funds have contributed somewhat to partners 
and stakeholders sharing and/or receiving literacy information and to literacy materials 
and tools being developed and made available to learners and stakeholders. As outlined in 
its management response to the 2002 Summative Evaluation recommendation to increase 
accessibility of projects and results, the program implemented a number of measures: 

• It began using HRSDC’s CSGC, expecting that “this new data tracking system capacities” 
would enable the capture of project results appropriate for dissemination; 

• It implemented a mechanism whereby a copy of all NLP-funded products is to be 
deposited at HRSDC’s library, which is accessible to the public and whose catalogue is 
available on-line via the National Adult Literacy Database (NALD); 

• It required that all NLS-funded project descriptions be posted on the NALD web-site; 

• It explored the feasibility of putting all products/materials on NALD as full text documents 
(full text versions of materials were put on NALD in full text on an ad hoc basis); 

• It supported the literacy database at the University of Alberta; 

• It funded a number of newsletters such as Connect and Literacy.ca and supported a new 
hteracy research journal to publicize materials and resources developed; and 

• It supported conferences as a vehicle for sharing information, such as the National Best 
Practices Workshop on Literacy. 

In addition, the NLP requests that program applicants specify in their project funding 
application how they intend to distribute or make available reports or other products resulting 
from their project. 

Some of the above-listed intended changes were however not fully implemented. 

Lirst, while the number of NLP-funded products (reports and tools) posted on NALD has 
increased, there are indications that some may still be missing (although the total number of 
NLP-funded products, and therefore the number of products missing on NALD, is not 
recorded by the program). Lor example, a search of the NALD database resulted in only 
three reports from Prince Edward Island projects, and no reports from the projects reviewed 
for the PEI provincial case study. An analysis of NALD data was conducted to examine 
changes over time in the availability of NLP-funded products on the NALD website and 
the number of times the products were accessed by users. There were 502 NLP-funded 
products and 1,047 non-NLP- funded products on NALD with publication dates 1990 to 
2004. Data shows that the number of NLP-funded products on NALD rose gradually since 
1990. Prior to 1996, there were fewer than 10 NLP-funded products in each year. Prom 
2001 to 2004, the number of NLP- funded products on NALD ranged between 22 and 87. 
NLP-funded products were also more likely to be downloaded than non NLP-funded 
products: for products published in 2003 and 2004 and posted on NALD, 80% of the 109 
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NLP-funded documents were downloaded at least once during the period 2003-2005 
compared to 56% of the 234 non-NLP-funded documents. 

Second, although the CSGC was adopted for managing NLP project information, the system 
has yet to be used for information dissemination purposes. The NLP does not have 
a mechanism for documenting and sharing lessons learned from the various projects it funds. 
Final project reports are filed, with no systematic process for informing other similar 
organisations of the lessons learned and good practices developed or tested as part of these 
projects. According to key informants from NLP, this sharing occurs on an ad hoc basis and 
is dependent on the NLP officers/managers’ level of familiarity with a particular file 
or organisation. 

Despite the above-mentioned limitations, 79% (out of 191) of funding recipients 
surveyed consider that the NLP have contributed a lot or somewhat to information 
dissemination and case study evidence shows that at the project level some amount of 
dissemination occurs. 

Seventy-three percent (out of 191) of funding recipients surveyed reported that NLP 
contributed somewhat or a lot to improved access to literacy research, materials and/or tools. 
The most frequently mentioned methods of NLP project results dissemination by funding 
recipients were: presentations at workshops and seminars (56% out of 167), mail/e-mail 
(49% out of 167), posted on websites (40% out of 167), and published (40% out of 167). 

Organisation case studies illustrated that improved access to literacy services and outreach to 
potential beneficiaries was promoted through the establishment of linkages with other 
organisations that ensure mutual referrals to their respective programs. Organisations also sit 
on each other’s Boards of Administration or project advisory committees. Existing programs 
were expanded to additional sites across the country enhancing dissemination of information. 
Train the trainers sessions also ensured wider capacity to deliver services. Examples of 
improved access and outreach from organisation case studies include the following: 

• The Winnipeg Foundation: Some programs, targeted at new immigrants, have been 
adjusted to begin after Ramadan, in order to increase participation rates. 

• The Further Education Society of Alberta: As part of NEP-funded projects, the 
organisation held the first train-the-trainers program in francophone family literacy in 
the country; 63 trainers and 15 practitioners from across the country received training. 

• Shilo Military Resources Centre: The foundational family training course funded by the 
NEP includes a recruitment module, which guides practitioners in promoting and 
stimulating community interest and participation in family literacy programs. 

Best Practices 

A few examples were found as part of this evaluation of best practices being exhibited 
in the literacy field. 



A best practice is a technique or methodology that, through experience and research, has proven to reliably lead to 
a desired result. Source: http://searchvb.techtarget.com/sDefinition/0„sid8_gci498678,00.html 
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Several projects examined as part of case studies focused on adapting successful existing 
approaches to particular regional realities (e.g. development of Workplace Training 
materials in French by Le College du savoir), making them more accessible to 
practitioners in remote areas (e.g. Canadian Automotive Repair and Service Council’s 
training in snowmobile repair and maintenance delivered through local trainers, mentors 
and coaches), or expanding capacity to deliver successful programs across the country 
(e.g. development of eight new Home Instruction for Parents of Preschool Youngsters - 
HIPPY sites across Canada). 



Tough Challenges: Great Rewards, the NLP-supported literacy strategy in PEI, was cited 
by several informants as the best example of a best practice in PEI. An inclusive approach 
and the avoidance of silos in developing and implementing projects were cited as key 
components of this successful strategy. 



Also, 71% (out of 191) of funding recipients surveyed reported that the NEP had 
contributed somewhat or a lot to sharing of best practices. Respondents who had been 
funded under the national projects stream were more likely to rate the NEP as 
contributing a lot to the sharing of best practices (67% out of 54) than respondents funded 
under the provincial/territorial projects stream (36% out of 1 14). 

Public Awareness and Learner Participation 

While an increase in program participation could not be measured in the absence of baseline 
and project performance data, some evidence was collected illustrating the NEP’s 
contribution to awareness raising. 

Seventy-four percent (74% out of 191) of funding recipients surveyed indicated that the NEP 
had contributed somewhat or a lot to greater awareness of literacy programs and services 
among end-users. Awareness raising/promotional event was one of the three most frequently 
mentioned (by 60% out of 181) results of NEP-funded projects by funding recipients survey 
respondents. 



The Winnipeg Eoundation inserted profiles of its NEP funded program Literacy for Life 
in its magazine Working Together, held a television campaign that included a spot for 
Eiteracy for Eife, promoted the program at a popular golf tournament, and placed an add 
in the Winnipeg Eree Press every second Saturday. 



Organisation case studies showed that public awareness is promoted through organisations’ 
promotion materials and websites, promotion campaigns, and word of mouth. 

However, a review of the NEP’s administrative database showed that funding for public 
awareness projects has decreased over time. The total value of the public awareness projects 
during the 1997/98 to 2001-2002 fiscal years was between $6.5 mi llion and $7.6 mi llion 
compared to $3.1 million to $5.8 mi llion from 2002-2003 to 2004-2005. The $3.1 mi llion 
approved in 2004-2005 represented 10% of the funding approved compared to approximately 
25% over the first five fiscal years. 
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Creation of Partnerships and Leveraging of Additional Funding and Support 

The development of strategic partnerships was a common feature of all organisations 
examined as part of organisation and project case studies, each having developed as part 
of their NLP-funded projects at least one partnership with other organisations either for 
leveraging of additional in-kind or financial resources for the project, to expand delivery 
sites, or to benefit from the partner’s areas of expertise (e.g. business, industry, labour). 

Seventy percent (out of 191) of funding recipients surveyed said the NLP had contributed 
somewhat or a lot to the creation of effective strategic partnerships. A high percentage (89% 
out of 99) reported that some or all of the partnerships created as part of their NLP-funded 
project continued after the project ended. Several provincial case study informants stressed 
how important the NLP contribution had been as a catalyst for other funders to come on 
board in the early days of the PEI literacy strategy implementation, and since then as the 
strategy has evolved. Several examples were cited of initiatives in PEI that were started with 
NEP funding and sustained through other funding sources. These included a summer tutoring 
program. Story Sacks, and specific programming within Workplace Education PEL 



NLP funding helped Operation Go Home strengthen its relationship with the David Smith 
and McHugh Centres. Through these partnerships, OGH strengthened its partnership with 
the Ottawa Catholic School Board, which agreed to provide a teacher on a full-time basis 
and expand the support provided to street-involved youth to earn either their high school 
diploma or high school equivalency program. 



Evaluation Question: Initial learner impacts - Does the NLP contribute to the achievement 
of stronger literacy skills or other impacts among learners? 



Several organisation representatives and practitioners consulted as part of project and 
organisation case studies have reported positive impacts on learners, in particular families, 
members of Aboriginal communities, and members of Erancophone minority language 
communities. These impacts were, in most cases, measured on an ad hoc basis by the funded 
organisation or by the literacy practitioners who benefited from the NEP-funded project and 
in turn assisted learners. Eeamer impacts were reported for fourteen of the twenty projects 
examined as part of project case studies. While not every project examined entailed impacts 
on learners, some examples recorded include improvement of studying and literacy s ki lls 
among youth and increased motivation to return to school, which led to completion of high 
school or school credits for several. Some parents went on to complete high school as a result 
of family literacy activities. Immigrants improved their ability to function effectively in every 
day situations and were prompted to seek further education. Seniors developed their ability to 
speak in groups, to use the Internet, and to write stories. A common thread across these 
examples is the increased confidence in learning and self-esteem gained by learners as part of 
their literacy activities. 

The majority of funding recipients (62% out of 191) perceived that the NEP had 
contributed a lot or somewhat to improve literacy skills of learners and to their increased 
access to community life. 
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When asked what they thought was the most important difference the NLP funded products or 
services they used had made for learners, 51 respondents to the survey of end-users 
(practitioners) provided a range of open-ended answers. The most frequently mentioned were: 
improved access to appropriate learning materials and services (19 mentions); increased 
self-esteem and self-confidence regarding literacy learning and general participation in 
community life (10 mentions); improved knowledge or awareness of their literacy needs and 
of the available hteracy resources and services to help them address these needs (9 mentions); 
and improved literacy skills and other transferable s ki lls (8 mentions). Several (12), however, 
felt that the NLP funded outputs didn’t have observable impacts on learners. 

In summary, in the absence of program data collection on learner impacts, there is some case 
study related anecdotal evidence, but no broad-based evidence of program impacts on 
learners. 



Evaluation Question: Initial capacity building - Does the NLP contribute to increased 
capacity among stakeholders? 



For the purpose of this evaluation, capacity was defined at two levels. At the 
organisational level, it was defined in terms of funding recipient organisations’ capacity 
to leverage additional support for project activities, and to increase the knowledge, 
experience, or expertise of their staff/ volunteers, as well as additional human resources, 
new procedures/protocols, etc. At the individual level, it was defined as the knowledge 
and skill development of literacy practitioners and other individuals involved in literacy 
service development, administration or delivery. 



According to practitioners who attended an NLP-funded training workshop, the training 
has literally changed their teaching strategies with individuals presenting learning 
disabilities and made them more effective in the classroom. 



According to key informants from NLP, capacity building of funded organisations is one 
of the key contributions of the program. In particular, the NLP has helped create, develop 
and maintain seven national organisations: the Movement for Canadian Literacy (MCL), 
the Federation canadienne pour I'alphabetisation en frangais (FCAF), Frontier College, 
Laubach Literacy Canada, ABC Canada, the National Indigenous Literacy Association 
(NILA), and NALD, as well as provincial literacy coalitions in each province. 

“Before the NLP was created, the literacy community was not well 
coordinated, had an underdeveloped voice, and conducted limited advocacy 
efforts. Since the NLP’s inception, a literacy strategy has been developed 
in every province and territory, and members of the literacy community have 
learned to work with federal and provincial governments.” (Key Informant 
from NLP). 

NLP funds have also encouraged, supported and enabled the creation of key partnerships 
and the development of networks that have increased the capacity and stability of the 
recipient organisations and leveraged additional financial and in-kind resources for their 
NLP-funded projects. 
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Practitioners who benefited from of an NLP-fiinded project felt that it provided a valuable 
opportunity to experiment with an alternative way of engaging First Nations students 
in learning activities that are culturally relevant and meaningful. 



Some examples of initial capacity building of funded organisations reported as part of 
organisation case studies include the following: 

• NLP funding has had a significant influence on the capacity of Operation Go Home 
to offer educational programming for youth dealing with homelessness. One major 
outcome of the project is that Operation Go Home was able to secure long-term and 
ongoing support from the Ottawa-Carleton Catholic School Board. 

• The Association of Workplace Educators of Nova Scotia reported that NLP funds provided 
for the 2006 Workplace Practitioners’ Institute enabled them to leverage a further $31,000 
from the provincial Department of Education, as well as in- ki nd support from more than 
twenty-five businesses. 

NLP funding gave the Eurther Education Society of Alberta (ELSA) more visibility, 
credibility and recognition at the provincial and national levels, thereby opening doors for 
further activities and partnerships and for wider implementation of its program. The 
multi-year contribution funding enabled ELSA to plan strategically for the long term, to build 
durable partnerships, and to ensure some continuity within its human resources. 

Moreover, a majority of funding recipients surveyed (82% out of 191) indicated that 
without the funding, their project would have been cancelled or deferred until funding 
could be found. Only 7% (14 out of 191) said the project would have gone ahead with 
a reduced scope. As shown in Table 4, approximately 80% of the surveyed funding 
recipients indicated that the NLP had increased their organisation’s capacity a lot (53% 
out of 191) or somewhat (27% out of 191). 



Table 4 

Perceived Contribution of the NLP to Organisationai Capacity Buiiding 


Extent to which the NLP contributed to increasing 
the funding recipient’s capacity 


Percent 

of Respondents 


Not at all 


2.6% 


A little 


5.8% 


Somewhat 


27.2% 


A lot 


52.9% 


Don't know/no response 


5.2% 


Not applicable 


6.3% 


Total 


100.0% 


Number of respondents 


191 
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While several examples of NLP impacts on capacity building of literacy practitioners were 
provided through organisation and project case studies, Table 5 shows that, when asked about 
the contribution to the development of qualified trainers or facilitators, only 59% (out of 191) 
of surveyed funding recipients indicated that the NLP had contributed a lot or somewhat; 
20% (out of 191) of surveyed funding recipients said that this type of outcome was not 
applicable to their project. 



Table 5 

Perceived Contribution of the NLP to Practitioner Capacity Buiiding 


Extent to which the NLS contributed to the deveiopment 
of quaiified trainers and faciiitators 


Percent 

of Respondents 


Not at all 


5.2% 


A little 


10.5% 


Somewhat 


26.2% 


A lot 


32.5% 


Don't know/no response 


5.8% 


Not applicable 


19.9% 


Total 


100.0% 


Number of respondents 


191 



The most frequently mentioned impacts of the funded projects on literacy practitioners by 
case studies respondents were their increased capacity to serve their clientele, most notably 
through partnership building and networking opportunities, as well as increased access to 
relevant training tools and best practices. Conferences and workshops, in particular, provided 
practitioners with opportunities for networking and for face-to-face, open discussions with 
their peers, thus offsetting the isolation some of them experienced working in remote, rural 
areas. Due to their enhanced skills, practitioners report increased self-confidence and being 
better able to meet their communities’ needs. Importantly, some projects increased the 
capacity of practitioners to attempt alternative approaches to learning. 

Some examples of initial capacity building of literacy practitioners reported as part of 
project and organisation case studies include the following: 

• Practitioners agreed that the assessment tool and workshops offered as part of a project 
on learning disabilities were well-researched, well-organized, and extremely useful 
resources. One practitioner described that using the tool has helped to lessen the 
frustration she often felt at the fact that she is not well trained in learning disabilities. 

• Practitioners who took part in a research project spoke highly of the value of being 
exposed to the academic world, learning about the design and process of a large 
research project, and the added credibility it gave their work and their organisation. 

• Practitioners who attended a workplace literacy training event reported that it gave 
them new tools, resources and best practices to apply with learners; a renewed desire to 
pursue further professional development and certification; and an awareness of new 
trends and research in the field. 
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Evaluation Question: To what degree can HRSDC report on the literacy models and 
programs targeting families in Official Language Minority Communities? 



The NLP administrative database shows that 16 projects were funded under the Action Plan 
for Official Languages (APOL) Family Literacy Initiative between 2003-2004 and 2005-2006, 
for a total of $1,878,980. Fourteen of them started in 2004-2005 and are intended to last until 
2008. All projects are national contributions, except for three grants. Projects cover all 
provinces and territories except Quebec, Manitoba and Nunavut. Funding was awarded to the 
Federation canadienne pour Talphabetisation enfrangais (FCAF), tasked with coordinating 
the work of the newly created national Reseau d’ experts en alphabetisation familiale, and to 
umbrella literacy organisations in each province/territory. Project descriptions indicate that 
funding was targeted towards the development of networks, organisational and programming 
capacity, research, and promotion of family literacy in each of the provinces and at the 
national level through the work of the FCAF. This is consistent with the focus intended by 
the Family Literacy Initiative. The NLP database does not, however, provide information on 
the progress or outcomes of the projects to date. 

One project case study was conducted on a project funded by the NLP under the Initiative, 
L'Alphabetisation familiale en frangais au Nouveau-Brunswick - une demarche globale 
ancree dans les communautes implemented by the Federation d' alphabetisation du Nouveau- 
Brunswick (FANB). The case study illustrated the relevance of NLP support for professional 
development of literacy practitioners in Francophone minority communities and showed 
some positive preliminary outcomes of this training in terms of improved techniques and 
training approaches by practitioners and administrators in the field of family literacy. 

It is, however, too early to draw conclusions on the success of this initiative, given that 
funded projects remain in the early stages of their implementation. 



3.4 Other Evaluation Issues - Turned Towards the Future 

Whether sufficient data was collected to inform an eventual summative evaluation: 

This issue was partially covered in the Performance Measurement section. 

An eventual summative evaluation would benefit from data on the products and, if still 
desirable, on end-users of the projects funded by the program. This data was not 
systematically collected and recorded by the program. 

Changes to logic model: In response to the 2002 evaluation recommendation to make 
achievement of NLP objectives more measurable by clarifying program definitions, 
objectives and targets, the Governance Network was tasked with revising the program logic 
model as part of the interim and summative evaluation strategy for the NLS. The document 
states that the revisions were made to ensure that it “better reflects its foundation as 
a partnership based model and therefore clearly identifies that the NLS is one of many 
players contributing to improving literacy skills of Canadians.” The context for this 
recommendation and subsequent change to the logic model stems for the program’s difficulty 



The Governance Network. National Literacy Secretariat Draft Interim and Summative Evaluation Strategy. Febmary 2005. 
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in demonstrating the impacts of its activities on literacy levels of Canadians due to its indirect 
link to literacy learners. 

The resulting revised logic model (presented in Appendix E) provides a clear picture of 
the linkages between program activities and intended outputs and outcomes, and defines 
outcomes that the program can be held accountable for within its field of federal 
government jurisdiction. However, while this revision of the program logic model 
constitutes a first step in facilitating measurement of objectives achievement, this change 
appears insufficient to fully address the 2002 evaluation recommendation because the 
program did not develop specific targets to guide measurement. 

Whether new data collection requirements and methodologies have been put into place 
to ensure that both short and long-term outcomes, including changes made since the 
2002 evaluation, can be measured: This issue was covered in the Performance 
Measurement section. 



3.5 Performance Regarding 2002 Management 
Commitments 

More transparent funding decisions: This issue was partially covered in the Approach to 
determining funding priorities sub-section. 

However, survey of funding recipients shows that approximately only a half (57% out of 
191) of the respondents were satisfied with the transparency of the proposal review 
process while 16% were very dissatisfied or dissatisfied. The majority of representatives 
consulted through organisation case studies declared not being aware of the proposal 
review criteria and process used for distributing funding by NLP. 

Increasing the accessibility of projects and project results: This issue was covered in the 
Accessibility and dissemination of tools and materials sub-section. 

Simplifying the funding application process: During the 2002 evaluation, smaller 
organisations and those less familiar with the NLS indicated that the NLP application process 
needs to be simplified and streamlined. In its management response, HRSDC stated that NLS 
was currently making the funding application available on the web and providing simpler 
instmctions for the process. In addition a more transparent process was being developed 
by putting in place calls for proposals. 

The intended changes were brought to the NLP application process. The application form 
was made available on the NLP website and all organisations contained in the NLP 
database were mailed an application package containing instructions on how to submit an 
application, bunded organisations consulted as part of case studies generally expressed 
satisfaction with the application process, describing it as streamlined and straightforward. 
Seventy-one percent (135 out of 191) of funding recipients surveyed said they were 
satisfied or very satisfied with the simplicity of the funding application process, while 
14% said they were dissatisfied or very dissatisfied. 
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Steps taken to increase the reliability and credibility of local research projects: The 2002 
evaluation found that, while major research projects conducted through the Social Sciences 
and Humanities Research Council and Statistics Canada ensure high quality and reduce the 
risk of being duplicative through peer review mechanisms, NLP-funded research ran the risk 
of duplicating work done in other regions/communities. In addition, more than one-half of 
the provincial and territorial key informants interviewed as part of the 2002 evaluation felt 
that there was insufficient program monitoring and accountability on the FPT funding stream 
to ensure the quality of products. Furthermore, the evaluation concluded that needs 
assessment projects funded under the FPT stream may lack the rigour necessary to be 
considered “applied research” and that the application of a formal peer review process for all 
research conducted would help increase quality and help reduce the risk of duplication. One 
recommended change was to implement a peer review process for the evaluation of research 
project funding applications. 

In response, HRSDC pointed out that “the research projects largely include local needs 
assessment and project evaluations which would not be duplicated since they are geared to 
specific populations and projects.” Nevertheless, in order to address the issue of quality and 
reliability, the NLS committed to put in place specific templates and/or protocols to be used 
uniformly for this type of research and to complete a review of “all the research projects that 
have been funded over the past five years to synthesize the information and look at improved 
review processes.” 

No evidence was found of specific templates and/or protocols put in place to ensure the 
quality and reliability of NLP-funded research projects. As intended, a review was published 
by Ralph St.Clair in September 2004 on NLP-funded research reports, entitled Building 
a Community: Reviewing National Literacy Secretariat Research Support 1998-2003. 
This report concluded that: 

“[t]he contributions of the NLS to literacy research in Canada have been 
extremely valuable. They have allowed a lot of research to take place that 
otherwise would not have been feasible, and have created a body of new 
knowledge. By tackling the issues of research currently facing most research 
funders and tightening the definition of their purpose and process, the NLS 
has the opportunity to support even more valuable endeavours in the 
future.”^^ 

The report highlighted a number of areas requiring improvement, including: 

• insufficient explanations of research method in proposals; 

• insufficient demonstration of previous work on the proposed research topics and how 
the proposed projects will contribute to the overall body of work; 

• insufficient thematic targeting of research projects approved, reducing their potential 
for deeper understanding of key issues; 



Ralph St.Clair. Building a Community: Reviewing National Literacy Secretariat Research Support 1998-2003. 
September 2004. 
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• absence of a policy on funding organisational reviews and needs assessments, such 
projects using resources without clear benefits for the wider field but being vital for the 
survival of particular organisations; and 

• absence of a policy on who conducts research, allowing consultants to complete the 
majority of the research at the expense of research capacity building of the community. 

No evidence was found that the NLP has implemented changes to its funding of research 
projects to address these conclusions or the 2002 evaluation recommendations. 

Increasing the usage of contributions: In response to a departmental directive and a 2002 
summative evaluation recommendation to increase the accountability of NLP funded 
projects, an informal guideline was established in 2002 for contributions to be used for 
funding given to national organisations (except for the National Indigenous Literacy 
Association) and for projects of $350,000 or more, with the aim of reaching a proportion of 
22% contributions. Table 6 outlining NLP administrative data shows that the proportion 
of NLP funding given through contributions has increased to 22% in 2004-2005, up from 5% 
in 2003-2004. However, the proportion of projects approved as contributions was only 11% 
in 2004-2005, representing approximately half of the NLP’s intended target. 





Table 6 

Projects And Amount Approved by Funding Mechanism and Fiscai Year: 

Percentage Distribution 




Fiscai Year 


Funding 

Mechanism 


1997- 

1998 


1998- 

1999 


1999- 

2000 


2000- 

2001 


2001- 

2002 


2002- 

2003 


2003- 

2004 


2004- 

2005 


Total Funding Approved 


Contributions 


21 .0% 


13.8% 


15.8% 


9.4% 


10.1% 


7.1% 


5.3% 


22.3% 


Grants 


79.0% 


86.2% 


84.2% 


90.6% 


89.9% 


92.9% 


94.7% 


77.7% 


Projects Approved 


Contributions 


6.9% 


0.8% 


0.9% 


0.4% 


0.4% 


1.1% 


4.5% 


1 0.8% 


Grants 


93.1% 


99.2% 


99.1% 


99.6% 


99.6% 


98.9% 


95.5% 


89.2% 



Key informants from NLP expressed concerns with the additional delays and workload 
imposed on NLP staff by the increased use of contributions: the contribution application 
process requires more detailed information and the approval process is longer. 

Developing a framework for a Pan-Canadian Literacy Strategy: In its Management 
Response to the 2002 summative evaluation, the NLS committed to developing a framework 
for a Pan-Canadian Literacy Strategy. This commitment followed the Government of 
Canada’s response to “Raising Adult Literacy Skills: The Need For A Pan-Canadian 
Response”, where it identified the NLS as a key mechanism through which the Canadian 
government would contribute to the concerted effort to address literacy issues in Canada. 
In October 2005, the Movement for Canadian Literacy, in consultation with six other 
National Literacy Organisations, issued its report “A 10-year Results-Based National 
Literacy Action Plan 2006-2016”. This report presented a concrete action plan, from the 
perspective of the non-government literacy sector, for addressing Canada’s literacy 
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challenges. This report was followed, in November 2005, by the report “Towards a Fully 
Literate Canada. Achieving National Goals through a Comprehensive Pan-Canadian Literacy 
Strategy” presented by the Advisory Committee on Literacy and Essential Skills to the 
Minister of State for Human Resources Development. This report endorsed the Action Plan 
presented by the seven national literacy organisations, stated that “a comprehensive 
Pan-Canadian Literacy Strategy does not yet exist”, and called upon the Federal Government 
“to take an immediate leadership role in the development of a comprehensive Pan-Canadian 
Literacy Strategy”, thereby reiterating the commitment made by the NLS in 2004. This report 
proposed a vision statement, goals, principles, and strategic objectives for the implementation 
of a Pan-Canadian Literacy Strategy. As such, this document could be construed as 
a “framework” for such a strategy. No evidence was found that the proposed framework or 
any of the reports’ recommendations were to date adopted or implemented. 
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4. Key Conclusions 



This section contains the key conclusions developed from the findings of the interim evaluation 
of the National Literacy Program (NLP). 



4.1 Relevance 

Program Consistency with Needs and Priorities 

Evidence shows that the NLP’s support to developing literacy awareness, programming and access 
across Canada is compatible with both the federal government and Human Resources and Social 
Development Canada (HRSDC’s) priorities pertaining to inclusive participation of all Canadians in 
our new learning society and knowledge-based economy. 

While the program does not address literacy stakeholders’ expressed need for long-term core 
funding of literacy programming, it provides funding for coordination and networking, awareness 
and promotion, and research, areas literacy stakeholders identified as needing federal government 
support. The allocation of NLP funds is compatible with the varied needs of the literacy 
community given its responsive partnership approach to setting funding priorities and the flexible 
definition of its five activity areas: Learning Materials, Access and Outreach, Coordination and 
Information, Public Awareness, and Research. Also, there appears to be few alternative sources 
of funding, in the areas examined, for literacy capacity building, coordination and program 
development. As such, the NLP plays a unique and necessary role. 

Relevance of Federal Government Role in Literacy 

Overall, evidence shows that a need remains for federal government intervention in the field of 
adult literacy, particularly as an initiator, promoter and supporter of national initiatives and 
innovative partnerships among various stakeholders. The NLP’s contribution to literacy appears 
to be compatible and complementary to the role played by provinces, this complementarity being 
ensured by the effective partnership approach adopted. In this sense, this program does not 
appear to be a candidate for realignment with the provinces. 



4.2 Implementation 

Program Utilization and Reach 

Some evidence shows that the NLP’s direct interventions reach a wide variety of stakeholders, 
particularly local literacy organisations, non-profit service organisations and indirectly through 
funded project outputs, literacy practitioners and learners, including youth, families, and seniors 
from various Canadian provinces. The program has taken a responsive approach to addressing 
local literacy needs and, as such, did not set specific targets for the reach of its activities. 
Furthermore, no data is being collected by the program on the actual end-users of NLP-funded 
activities, thereby preventing an accurate assessment of its reach. 
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Program Implementation, Efficiency and Effectiveness 

Program eligibility criteria are clearly outlined in the application package and neither NLP 
representatives nor funding recipients expressed having particular issues with this component of 
program implementation. 

In the short-term, the NLP aims to achieve: 1) enhanced literacy research in identified priority areas; 
2) improved dissemination and sharing of information among literacy stakeholders; 3) increased 
availability of adult literacy materials and tools by learners and practitioners; 4) greater awareness of 
literacy programs, services and issues among targeted sectors; 5) improved access to literacy skills by 
Canadians with low literacy skills; and 6) improved strategic partnerships involved in literacy issues. 
NLP’s approach appears to be appropriate for achieving these outcomes given that it funds projects in 
each of the first five categories. Achievement of the sixth is ensured by the program’s requirement 
that funding applicants secure additional project funding and support from partners. Furthermore, the 
NLP has adopted a responsive approach to establish more precise project selection criteria, within the 
first five outcome categories. These criteria are based on local needs and priorities and established in 
collaboration with each province/territory. While NLP and PEI government representatives find this 
approach to establishing project selection criteria appropriate, given jurisdictional realities, funding 
recipients from organisation case studies were generally unaware of the process, thereby indicating 
that the program could be more explicit in its information regarding the program. In the same way, 
only 57% (out of 191) of funding recipient survey respondents expressed satisfaction with the 
transparency of the proposal review process and 54% with the timeliness of the funding approval 
process. 

Performance Measurement 

Although the NLP collects information that could be used to assess the performance of the projects 
it funds, it does not currently have a specific set of performance measures against which the results of 
its activities and funded projects could be assessed, thereby limiting NLP’s performance 
measurement and reporting capacity. Furthermore, the program does not have a system in place 
to store and use project performance information. In the absence of such a system, the NLP has 
limited ability to measure and report on whether it is achieving its objectives. The decision not to 
pursue initial steps to implement a performance measurement system was attributed in part to Senior 
Management’s desire to focus on the new ALLESP, for which a performance measurement system 
is being developed. 

No evidence was found that performance evidence is being used for program or management 
decision-making purposes on an ongoing basis. 



4.3 Success 

Achievement of NLP Objectives 

Evidence shows that the NEP is contributing to meeting its objectives of supporting the production 
and utilisation of quality and useful literacy tools and materials, improvements in accessibility and 
dissemination of literacy services and resources, the application of literacy best practices, and the 
creation of strategic partnerships. 
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Observed NLP project outputs included a variety of literacy tools and materials (e.g. training 
manuals, assessment tools, online training programs, simple language learner materials, bulletins, 
etc.), research reports, partnerships with other organisations, and awareness raising and 
promotional events and products. 

A majority of project outputs are considered good quality and highly useful by practitioner end-users 
consulted as part of case studies, including culturally relevant training materials that make use of 
concrete examples and clear and simple language, while some products are seen by literacy experts to 
present low quality and relevance. The 2002 evaluation had identified limitations in the reliability and 
credibility of local research. An NLS-commissioned research reviewing NLP-funded research reports 
from 1998 to 2003 concluded that the NLS had contributed substantially to literacy education 
in Canada through its research function but that improvements could still be made, among others, in 
detailing the research methodology and drawing linkages to other research work. Similar concerns 
were expressed by some experts with regards to a sample of research reports examined. No evidence 
was found that the NLP’s approach to funding research projects was modified following this study or 
to address the 2002 evaluation recommendations. 

However, end-user survey and case study results showed that NLP-funded products are being 
used by a wide variety of individuals, including literacy professionals and learners, for a wide 
range of purposes including improving current literacy training practices and developing new 
literacy training programs and tools. Strategic partnerships, for leveraging additional funding 
and/or for sharing expertise and resources, are also a key feature of all projects examined. 

There was weaker evidence of NLP-funded activities contributing to increased public awareness and 
learner participation, in part due to the absence of data on the number and profile of project end-users. 
This lack of data can be attributed to the fact that the NLP does not have a performance measurement 
system in place but also to the fact that the NLP only has indirect access to project end-users and that 
project funding recipients are reluctant to disclose end-user contact information for confidentiahty 
reasons. 

Some limitations were also found in the NLP’s mechanisms to promote accessibility and 
dissemination of tools and materials. While the program expects projects to disseminate their 
project results and encourages posting of funded products on NALD and various provincial 
literacy websites, evidence shows that this is not systematically done and the NLP does not have 
a formal mechanism for rolling up the lessons learned and best practices developed by its funded 
projects for the benefit of all literacy stakeholders. 

Initial Learner Impacts 

In the absence of specific data on the learners who benefited from NLP-funded products, anecdotal 
evidence was collected from funded organisations and from literacy practitioners who witnessed first 
hand the impacts of these products on a variety of individuals. While not every project examined 
entailed impacts on learners, some examples recorded include improvement of studying and literacy 
skills among youth and increased motivation to return to school, which led to completion of high 
school or school credits for several. Some parents went on to complete high school as a result of 
family literacy activities. Immigrants improved their ability to function effectively in every day 
situations and were prompted to seek further education. Seniors developed their ability to speak in 
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groups, to use the Internet, and to write stories. A common thread across these examples is the 
increased confidence in learning and self-esteem gained by learners as part of their literacy activities. 

Initial Capacity Building 

Stronger evidence was collected of the NLP’s contribution to capacity building of funded 
organisations and of the literacy field in general, and to professional development of literacy 
practitioners. The NLP’s support to national literacy organisations, its influence in promoting new 
emerging areas such as workplace and family literacy, and its support and encouragement for the 
creation of strategic partnerships have been recognized by NLP managers, PEI government 
representatives, as well as funding recipients and their project partners. A majority of funding 
recipients surveyed also reported that their project would not have been implemented without NLP 
funding. 

Examples of positive impacts of NEP funding on funded organisations include the ability to 
secure long-term and ongoing funding and support from other sources; increased visibility, 
credibility and recognition of the organisation at the provincial and national levels, thereby 
opening doors for further activities and partnerships, and for wider implementation of its 
programs; and stabilisation of the organisation’s finances, thereby allowing longer-term planning 
and better retention of its human resources. 

The most frequently mentioned impacts of the funded projects on literacy practitioners were their 
increased capacity to serve their clientele, most notably through partnership building and 
networking opportunities, as well as increased access to relevant training tools and best practices. 
Conferences and workshops, in particular, provided practitioners with opportunities for 
networking and for face-to-face, open discussions with their peers, thus offsetting the isolation 
they experienced working in remote, rural areas. With the enhancement of their skills, 
practitioners consulted as part of case studies report that they have enjoyed increased confidence 
and are better able to meet their communities’ needs. 

Targeting of Families in Official Language Minority Communities 

Sixteen projects were funded under the Action Plan for Official Eanguages (APOE) Eamily 
Eiteracy Initiative between 2003-2004 and 2005-2006. However, it remains too early to draw 
conclusions on the success of this initiative, given that funded projects remain in the early stages 
of their implementation. Project descriptions indicate that funding was targeted towards the 
development of networks, organisational and programming capacity, research, and promotion of 
family literacy in each of the provinces and at the national level through the work of the ECAE. 
This is consistent with the focus intended by the Eamily Eiteracy Initiative. 



4.4 Other Evaluation Issues - Turned Towards the Future 

Whether Sufficient Data Was Collected to Inform an Eventual Summative Evaluation 

An eventual summative evaluation would benefit from data on the products and, if still desirable, 
on end-users of the projects funded by the program. This data was not systematically collected 
and recorded by the program. 
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Changes To Logic Model 



Intended revisions have been made to the program logic model, better reflecting the linkages 
between its activities and expected outcomes and proposing outcomes that better reflect the 
program’s levels of accountability. However, in order to fully meet the intent of these changes, 
which is to make achievement of NLP objectives more measurable, the program would have 
needed to approve performance indicators and targets for each of the expected outcomes. 

Requirements and Methodologies in Place to Ensure Measurability of Outcomes 

As mentioned, a performance measurement system is not in place to ensure measurability of the 
program outcomes. This system would have required detailed performance measures and 
a mechanism for systematically collecting information on project outputs and, if possible and 
desirable, on end-users. 



4.5 Performance Regarding 2002 Management 
Commitments 

More Transparent Funding Decisions 

Feedback collected from funding recipients indicates that the process for making funding decisions is 
unknown to many and an aspect of the program’s implementation gamering one of the lowest 
satisfaction ratings. 

Increasing the Accessibility of Projects Results 

While the NLP has put in place a number of measures to increase accessibility of project results, 
posting of project outputs on NALD and other provincial websites is not systematically done and 
the NLP does not have a formal mechanism for rolling up the lessons learned and best practices 
developed by its funded projects for the benefit of all literacy stakeholders. 

Simplifying the Funding Application Process 

Improvements were brought to the NLP application process, namely the posting of application 
forms and instruction on the NLP website, which appear to have addressed a majority of funding 
recipients’ concerns in this respect. Seventy-one percent (out of 191) of funding recipients 
surveyed said they were satisfied or very satisfied with the simplicity of the funding application 
process, while 14% said they were dissatisfied or very dissatisfied. 

Steps Taken To Increase the Reliability and Credibility of Local Research Projects 

Although a study was commissioned by the NLP to examine the research reports funded between 
1998 and 2003, no evidence was found that the NLP has implemented changes to its approach in 
funding research projects to increase their reliability and credibility. 
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Increasing the Usage of Contributions 



The NLP has established an informal guideline for contributions to be used for funding given to 
national organisations (except for the National Indigenous Literacy Association) and for projects of 
$350,000 or more, with the aim of reaching a proportion of 22% contributions. The proportion 
of contribution projects has increased to 11% in 2004-2005, while the proportion of contribution 
funding has increased to 22%. However, there is no evidence that the shift operated towards 
increased use of contributions has had the intended effect of increasing project accountability. It has 
however increased the administrative burden on both NLP staff and funding recipients, while 
increasing the length of the project approval process. 

Developing a Framework for a Pan-Canadian Literacy Strategy 

A framework for a pan-canadian literacy strategy was presented in November 2005 as part of the 
report “Towards a Fully Literate Canada. Achieving National Goals through a Comprehensive 
Pan-Canadian Literacy Strategy”, thereby addressing the NLS commitment made in its Management 
Response to the 2002 summative evaluation. No evidence was found, however, that the proposed 
framework or any of the reports’ recommendations were to date adopted or implemented. 
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Appendix A 
Evaluation Matrix 




Interim Evaluation of the National Literacy Program 



‘Partner” refers to those individuals or groups that the NLS works with on a collaborative basis, such as provincial governments and national literacy organisations. “Stakeholder” 
refers to those individuals or groups that have an interest in NLS activities and outcomes. This could include literacy experts, researchers, and practitioners. 




Evaluation Matrix (continued) 
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Learner” refers to those individuals who have the potential to benefit from literacy services. 
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Evaluation Matrix (continued) 
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Evaluation Matrix (continued) 
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Appendix B 
Project Case Studies 
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Profile of Cases selected for Project Case Studies (continued) 
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Appendix D 
Expert Panel Products 



List of Products Reviewed by Expert Panei 


No. 


Product 

Name 


Type of 
Product 


Description 


Funded 

Organisation 


Project Titie 


1 


Alpha au 
boulot 


Tool 


Training manual in 
French for workplace 
literacy trainers 


Le College du 
savoir, Ontario 


Partenaires pour 
I’action en milieu 
de travail 


2 


Defendre nos 
droits 


Tool 


Simple language 
information document 
in French on learners’ 
rights 


Regroupement des 
groupes populaires 
en alphabetisation 
du Quebec, 
Quebec 


Pratiques et 
politiques: enjeux 
internes et externes 


3 


Guide de la 
personne 
tutrice 
benevolo 


Tool 


Training manual in 
French for volunteer 
literacy trainers 


The Frontier 
College, Ontario 


Frontier College: 

An Integrated 
Community Literacy 
Resource 


4 


Storysacks 


Tool 


Guide and handout 
materials for 
implementation of 
a literacy training 
workshop 


Nunavut Literacy 
Council, Nunavut 


Nunavut Family and 
Community Literacy 
Development Project 


5 


Online 

Mentoring 

Tool 

www.LBSorac 

titionertrainino 

.com 


Tool 


Website offering various 
professional 
development resources 
for LBS practitioners 


Ontario Assoc, of 
Adult & Continuing 
Edu. School Board 
Administrators, 
Ontario 


Measuring Quality in 
Mentorship for LBS 
Practitioner Training 
Online 


6 


Connecting 
Generations 
of Women 
Through 
Literacy 


Tool 


Basic language training 
program guide targeting 
Aboriginal women 


North West 
Regional College, 
Saskatchewan 


Connecting 
Generations of 
Women Through 
Literacy 


7 


Learning 

Challenges 

Assessment 

Tool 


Tool 


Assessment resource 
for the presence of 
learning challenges in 
adult literacy learners 


Adult Basic 
Education 
Association 
(ABEA), Ontario 


You’ve Been Trained 
in Learning 
Disabilities- 
Now What? 


8 


Bulletins for 
LBS 

Practitioners 
and Learners 


Tool 


A series of five bulletins 
for learners and 
practitioners of LBS 
programs in Ontario 


Ontario Literacy 
Coalition, Ontario 


Workforce Literacy 
Resources: Bulletins 
for LBS Practitioners 
and Learners 
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List of Products Reviewed by Expert Panei (continued) 


No. 


Product 

Name 


Type of 
Product 


Description 


Funded 

Organisation 


Project Titie 


9 


Informal 
Learning 
Practices of 
Adults with 
Limited 
Literacy 
Skills, A 
Research 
Summary 


Research 

Report 


Summary of 
ethnographic research 
on informal learning 
practices of adults with 
low literacy skills 


Partnership in 
Learning, Ontario 


Literacy Practices 
and Media 

Perceptions of Adults 
with Low Literacy 
Skills 


10 


Strategies 
for Success. 
Research 
Results 


Research 

Report 


Results of a survey of 
Manitoba literacy 
practitioners on their 
experience with adult 
learners with learning 
disabilities 


LiteracyWorks, 

Manitoba 


Strategies for 
Success 


11 


Exploring 
Learners’ 
Perspectives 
on Progress 


Research 

Report 


Report on a research 
exploring learners’ 
experiences and 
understanding of 
progress 


Parkdale Project 
Read, Ontario 


Exploring Indicators 
that Support 
Successful 
Transitions to 
Greater 
Independence: 

A Research Project 
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Appendix E 
NLP Logic Model 
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